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ABSTRACT 


TAC  THUNDER  is  the  Air  Force's  premier  campaign  level  model.  Perhaps  the 
most  important  input  to  TAC  THUNDER  is  the  user-defined  apportionment  of  available 
aircraft.  This  apportionment  defines  the  percentage  of  each  type  of  aircraft  that  are 
assigned  to  specific  missions.  An  optimal  apportionment  represents  the  best  use  of 
aircraft  available  to  the  theater  commander.  Campaign  outcomes  using  an  optimal 
allocation  are  a  powerful  tool  for  demonstrating  that  no  superior  non-material  solution 
exists  as  far  as  aircraft  employment  is  concerned.  That  is,  an  optimal  allocation  addresses 
Congress's  concern  that  US  military  commanders  fully  exploit  current  weapons  systems 
before  acquiring  replacement  systems. 

To  effectively  compare  different  sets  of  available  aircraft,  it  is  necessary  to  find 
the  apportionment  that  maximizes  the  effectiveness  of  each  set  of  aircraft.  This  research 
effort  uses  an  unclassified  scenario  to  show  how  an  analyst  can  use  response  surface 
methodology  (RSM)  techniques  to  estimate  the  relationship  between  aircraft 
apportionment  and  campaign  outcomes. 

The  optimal  apportionment  for  one  phase  of  a  hypothetical  conflict  was  found  by 
using  a  steepest-gradient  search  of  the  constrained  response  surface.  The  results  are 
illustrated  by  illuminating  charts  showing  the  various  relationships  between  the  number 
of  aircraft  in  the  theater  and  measures  of  effectiveness  such  as  PLOT  movement  and 
attrition.  The  results  show  close  air  support  and  interdiction  missions  to  be  highly 
effective  in  this  phase  of  the  scenario.  The  effect  of  the  enemy's  air-to-air  probability  of 
kill  (AAPk)  and  surface-to-air  probability  of  kill  (SAPk)  on  the  MOEs  provides  insight 
into  the  robustness  of  the  solution  in  terms  of  these  operational  uncertainty  parameters. 
This  type  of  sensitivity  analysis  also  provides  useful  information  on  how  the  TAC 
THUNDER  algorithms  interact  to  simulate  the  combat  environment. 
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I.  Introduction 


It  is  absolutely  necessary  to  study  new  systems  in  the  context  of  the 
environment  in  which  they  will  participate.  We  must  measure  the  inter¬ 
dependencies  that  are  becoming  a  bigger  and  bigger  part  of  a  system's 
contribution  to  the  overall  campaign  (Smith,  1992;  13) 

In  this  era  of  declining  defense  expenditures  and  increasingly  expensive  research 
and  development  programs,  the  U.S.  must  focus  its  R&D  dollars  on  those  programs  that 
add  the  most  capability  per  tax  dollar  spent.  Computer  simulations  of  future  battlefields 
provide  an  accepted  firamework  for  comparing  alternative  weapon  systems.  In  evaluating 
such  alternatives,  the  effectiveness  of  each  weapon  system  must  include  its  own 
capabilities,  as  well  as  its  contribution  to  the  combat  effectiveness  of  the  overall  force. 

The  Aeronautical  Systems  Center  (ASC)  uses  a  computer  simulation  called  TAC 
THUNDER  to  simulate  a  theater-level  conflict.  ASC  uses  TAC  THUNDER  to  evaluate 
the  effectiveness  of  various  aircraft  designs  and  modifications.  By  simulating  a  conflict 
repeatedly,  analysts  can  measure  the  effect  of  changing  the  number  and  type  of  aircraft  on 
the  outcome  of  the  hypothetical  war.  From  a  force  analysis  point  of  view,  the  results  of  a 
simulated  war  would  ideally  depend  on  only  the  characteristics  of  the  forces  involved. 

Not  surprisingly,  the  manner  in  which  forces  are  employed  has  a  tremendous  impact  on 
the  outcome  of  the  war. 

Current  acquisition  regulations  recognize  that  a  weapon  system's  effectiveness 
depends  on  force  employment  doctrine.  The  justification  for  a  new  weapon  system  must 
show  that  the  deficiency  the  system  is  designed  to  eliminate  caimot  be  eliminated  by 
using  current  systems  more  effectively.  Using  current  systems  more  effectively  is 
referred  to  as  a  "non-material  solution".  The  non-material  solution  issue  could  be 
resolved  if  an  optimal  force  employment  strategy  for  both  sides  could  be  found. 

However,  theater-level  combat  models  are  generally  very  large  and  complex.  To 
simultaneously  optimize  the  use  of  ground,  sea,  and  air  forces  on  both  sides  of  a  large 
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simulated  conflict  is  computationally  intractable.  ASC/XREC,  the  campaign  analysis 
office  for  ASC,  sponsored  this  thesis  in  an  effort  to  develop  a  method  to  determine  if  non¬ 
material  solution  alternatives  are  available  for  Air  Force  weapon  system  proposals. 
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11.  Background 


Introdwttion 

This  chapter  provides  a  background  on  the  material  used  in  this  thesis.  It  is 
intended  to  familiarize  the  reader  with  the  diverse  topics  covered  and  provide  references, 
via  the  bibliography,  for  those  readers  interested  in  additional  in-depth  information.  The 
major  topics  covered  are  the  air  campaign,  air  apportionment,  TAC  THUNDER,  and 
response  surface  methodology  (RSM).  The  air  campaign  is  the  primary  focus  of  the 
computer  simulation  TAC  THUNDER.  One  of  the  major  user  inputs  to  TAC  THUNDER 
is  the  apportionment  of  air  power  to  various  mission  classes.  RSM  is  the  set  of 
techniques  used  to  investigate  the  functional  relationship  between  the  air  apportionment 
and  the  outcomes  of  the  simulated  campaign.  Applying  optimization  methods  to  this 
fimctional  relationship  can  identify  the  best  air  apportionment  for  a  given  scenario  and 
measure  of  effectiveness. 

Air  Campaign 

AFM 11-1  defines  the  air  campaign  as  "A  connected  series  of  operations 
conducted  by  air  forces  to  achieve  joint  force  objectives  within  a  given  time  and  area  of 
operations"  (AFM  1-1, 1992).  In  order  to  execute  this  series  of  operations,  the  air 
commander  must  decide  where  to  focus  his  efforts.  He  must  apportion  his  aircraft 
between  the  various  missions  they  can  perform.  For  example,  air  superiority  aircraft  can 
be  used  offensively  or  defensively.  Similarly,  aircraft  that  deliver  munitions  can  do  so  to 
a  variety  of  target  classes  such  as  front  line  troops,  reserve  troops,  bridges,  and  airfields. 
The  percentage  of  available  aircraft  assigned  to  perform  each  of  these  missions  is  referred 
to  as  the  aircraft  apportionment. 
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It  is  vitally  important  that  the  commander  use  his  resources  most  effectively  to 
accomplish  his  objectives.  Technical  superiority  is  not  sufficient  to  guarantee  victory 
(Warden,  1989). 

Air  Apportionment 

"Allocation  strategy  is  the  most  powerful  driver  of  all  the  factors  which  determine 
the  outcomes  of  campaigns  and  wars"  (Smith,  1992;3).  Air  forces  are  allocated  through 
the  air  apportionment.  It  assigns  a  percentage  of  available  aircraft  to  specific  missions 
and  can  change  from  day  to  day. 

The  missions  defined  in  TAG  THUNDER  (version  5.9)  are: 

-  Close  air  support  (CAS)  -  attack  units  engaged  in  direct  fire  combat. 

-  Battlefield  air  interdiction  (BAI)  -  attack  support  and  2nd  echelon  units. 

-  Air  interdiction  (INT)  -  attack  the  enemy's  logistic  facilities,  facilities,  choke 
points,  transshipment  points,  rear-area  units,  or  supply  trains. 

-  Offensive  counter-air  (OCA)  -  attack  the  enemy’s  capability  to  generate  sorties.  These 
missions  exclusively  attack  air  bases. 

-  Strategic  targets  interdiction  (STI)  -  attack  the  enemy's  strategic  targets. 

-  Fighter  sweep  (FSWP)  -  an  offensive  air-to-air  mission,  attacks  enemy  aircraft  that  are 
operating  on  their  own  side  of  the  PLOT  during  a  period  when  fiiendly  ground  attack 
aircraft  are  operating  in  the  vicinity. 

-  Defensive  counter-air  (DCA)  -  aircraft  sit  on  the  ground  and  wait  for  the  enemy's  OCA 
or  INT  attacks.  A  DCA  aircraft  flies  only  in  response  to  an  attack.  If  it  takes  off  and  fails 
to  intercept  the  enemy,  it  acts  exactly  like  a  BARCAP  aircraft. 

-  Barrier  combat  air  patrol  (BARCAP)  -  aircraft,  while  on  station,  patrol  a  designated 
area  and  attempt  to  intercept  any  enemy  aircraft  that  attempt  to  pass. 


2-2 


-  Over-FLOT  defensive  counter-air  (ODCA)  -  aircraft  sit  on  the  ground  and  wait  for  a 
certain  type  and  size  enemy  flight  to  take  off.  An  ODCA  aircraft  then  flies  up  to  a  user- 
defined  distance  into  enemy  territory  to  attack  the  flight. 

-  Stand-off  lethal  air  defense  suppression  (SSUP),  stand-off  air  defense  jamming 
(SJAM),  close-in  lethal  air  defense  suppression  (CSUP),  and  close-in  air  defense 
jamming  (CJAM)  -  aircraft  orbit  the  battlefield  and  are  available  to  support  other 
missions  that  are  threatened  by  enemy  air  defenses.  The  two  primary  differences  between 
close-in  and  stand-off  missions  lie  in  the  manner  in  which  targets  are  assigned  and 
generated  and  the  aircraft  orbit  location  with  respect  to  the  PLOT.  As  far  as  mission 
execution,  there  is  no  difference. 

-  Direct  lethal  air  defense  suppression  (DSEAD)  aircraft  attack  air  defense  sites  within 
a  corridor.  Aircraft  will  attack  the  assigned  air  defense  sites  for  a  given  amount  of  time 
before  the  corridor  becomes  active. 

-  Reconnaissance  (RECCE)  -  collects  intelligence  about  the  enemy. 

-  Lethal  air  defense  suppression  escort  (ESUP)  -  aircraft  accompany  missions  to 
provide  lethal  suppression  of  enemy  air  defenses. 

-  Air  defense  jamming  escort  (EJAM)  -  aircraft  accompany  mission  to  provide 
jamming  suppression  of  enemy  air  defenses. 

-  Air-to-air  escort  (AIRESC)  -  aircraft  accompany  missions  to  provide  air-to-air 
protection. 

-  Reserve  (RES)  -  aircraft  being  held  for  possible  future  use. 

Not  every  scenario  requires  every  mission  type.  For  example,  strategic  target 
interdiction  (STI)  is  used  only  when  there  are  strategic  targets  in  the  theater.  These  targets 
are  usually  more  political  in  nature  and  do  not  directly  impact  the  capabilities  of  the 
forces  fighting.  Some  scenarios  require  that  100%  of  the  aircraft  be  assigned  to  non¬ 
reserve  missions. 
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TAC  THUNDER 


TAC  THUNDER  is  a  piston  driven,  theater-level  combat  model.  It  uses  an 
aggregated,  deterministic  ground  war  and  a  detailed,  stochastic  air  war  to  simulate  a 
theater-level  conflict  (TAC  THUNDER  Analyst's  Manual,  1992).  It  models  ground  units 
at  the  regiment  and  division  level,  consistent  with  the  Army's  FORCEM  model.  Aircraft 
sorties  are  modeled  individually. 

By  simulating  the  same  theater-level  campaign  imder  different  conditions,  such  as 
the  addition  of  new  equipment  (or  the  substitution  of  one  set  of  equipment  for  another), 
TAC  THUNDER  provides  a  means  to  assess  the  effect  of  those  changes  on  the  campaign 
objectives.  The  advantage  of  a  theater-level  model  over  smaller,  higher  fidelity  models 
is  that  it  measures  the  impact  of  each  alternative  in  terms  of  the  overall  theater  campaign. 

Each  TAC  THUNDER  scenario  requires  over  sixty  data  files  to  define  the  size, 
shape,  and  geography  of  the  theater  as  well  as  the  forces  and  objectives  of  both  sides. 

The  model  uses  this  input  data  to  assign  aircraft  to  specific  targets  on  each  day  of  the 
war.  Most  of  the  data  files  are  simply  descriptive,  but  the  apportionment  of  available 
aircraft  into  TAC  THUNDER-defined  missions  is  a  critical  input.  Since  the 
apportionment  determines  the  allocation  of  air  resources,  an  unacceptable  campaign 
outcome  might  be  due  to  an  inappropriate  apportionment  rather  than  to  inadequate 
aircraft  capabilities.  To  ensure  that  the  outcomes  reflect  the  capabilities  of  the  aircraft, 
the  apportionment  file  must  assign  the  right  aircraft  to  the  right  missions. 

In  TAC  THUNDER,  aircraft  are  assigned  to  mission  classes',  a  percent^e  of  each 
mission  class  is  then  apportioned  to  specific  missions.  For  example,  F-15  and  F-14 
aircraft  are  grouped  together  to  form  the  air  superiority  mission  class.  The  air 
apportionment  assigns  a  percentage  of  air  superiority  aircraft  to  air  escort,  fighter-sweep, 
and  barcap  missions.  Other  mission  classes  include  multi-role,  deep  strike,  and  ground 
support. 
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TAG  THUNDER  is  written  in  SIMSCRIPT  II.5.  SIMSCRIPT  is  a  structured 
simulation  language  which  is  designed  to  be  nearly  self-documenting.  TAG  THUNDER 
requires  500  megabytes  of  hard  drive  space  and  can  be  run  on  any  machine  that  runs 
SIMSGRIPT II. 5.  A  SUN  workstation  or  DEG  station  is  required  to  support  the  terminal 
graphics  used  in  TAG  THUNDER'S  situation  map  and  grapher. 

Response  Surface  Methodology  (RSIVD 

RSM  is  a  structured  set  of  techniques  for  empirical  model  building  and  model 
exploitation  (Box  and  Draper,  1987;1).  These  techniques  include  design  of  experiments, 
least  squares  regression,  and  optimization.  Several  good  texts  exist  to  assist  readers  who 
wish  more  information  on  RSM  (Box  and  Draper,  1987),  design  of  experiments 
(Schmidt,  1991),  or  applied  regression  (Draper  and  Smith,  1981). 

RSM  seeks  to  relate  a  response  or  output  variable  to  the  values  of  a  number  of 
predictors  or  input  variables.  The  prediction  of  the  response  to  various  input  variables  is 
called  a  response  surface.  Since  a  simulation  is  a  model  itself,  the  response  surface  of  a 
simulation  is  sometimes  referred  to  as  a  metamodel  (a  model  of  a  model). 

Empirical  models  attempt  to  explain  how  the  inputs  relate  to  the  output  of  a 
process.  Empirical  model  building  estimates  the  response  in  an  area  of  immediate 
interest.  The  true  response  function  is  unknown  but  it  is  assumed  that  it  can  be  locally 
approximated  by  a  polynomial  or  some  other  type  of  function.  This  polynomial  can  be 
conceptualized  as  an  "analytic  French  curve."  It  approximates  the  true  response  function 
over  a  small  experimental  region. 

If  the  air  apportionment  can  be  reduced  to  a  reasonable  number  of  factors,  then 
standard  RSM  techniques  can  convert  a  set  of  simulation  runs  conducted  in  accordance 
with  an  experimental  design  into  a  metamodel  of  the  simulation. 
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Sequential  experimentation  is  a  structured  method  of  scientific  investigation.  This 
research  used  sequential  experimentation  as  the  framework  to  guide  the  application  of 
RSM  techniques.  Sequential  experimentation  consists  of  an  iterative  cycle  of 
investigation:  CONJECTURE,  DESIGN,  EXPERIMENT,  and  ANALYSIS.  Figure  2.1 
illustrates  this  cycle.  Conjecture  is  the  hypothesis  that  the  analyst  wants  to  test.  Design 
is  the  synthesis  of  a  suitable  experiment  to  test,  estimate,  and  develop  a  current 
conjectured  hypothesis.  Analysis  is  the  treatment  of  the  experimental  results,  leading 
either  to  verification  of  a  postulated  model  and  the  working  out  of  its  consequences,  or  to 
the  forming  of  a  new  or  modified  conjecture  (Draper,  N.  R.  and  H.  Smith;8). 


Figure  2.1  The  iterative  nature  of  experimentation,  (from  Empirical  Model- 
Building  and  Response  Surfaces  by  Box  and  Draper) 

The  conjectures  for  a  particular  experiment  depend  on  the  experimenter's 
understanding  of  the  process  under  study.  In  the  first  stage  of  studying  a  process,  the 
conjectures  focus  on  identifying  WHICH  variables  are  important.  After  identifying 
WHICH  are  the  important  variables,  the  conjectures  focus  on  identifying  HOW  the 
response  behavior  relates  to  the  important  input  variables.  This  stage  is  often  called 
empirical  model  building  since  the  product  is  an  empirical  relationship  between  inputs 
and  outputs.  Finally,  the  conjectures  focus  on  WHY  the  process  responds  the  way  it 
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does.  This  stage  is  often  called  mechanistic  model  building  because  the  product  is  an 
understanding  of  the  underlying  mechanisms  of  the  process. 

The  conjecture  establishes  the  information  required  by  a  particular  experiment. 
The  design  should  be  of  sufficient  resolution  to  verify  or  disprove  the  conjecture.  A 
design  is  said  to  alias  or  confound  two  factors  if  the  design  does  not  allow  them  to  be 
distinguished  from  one  another. 

A  design  is  said  to  be  of  resolution  k  if  all  n^  order  terms  are  not  aliased  with  any 
terms  lower  than  order  k-n,  where  n  <  k.  For  example,  in  an  Rju  design  (k=3),  the  l^t 
order  (n=l)  terms  in  a  resolution  design  are  aliased  with  order  or  higher  terms  (k- 
n=2).  Thus,  Rjii  designs  are  considered  1^*  order  designs  because  no  1®^  order  terms 
(main  effects)  are  aliased  with  one  another.  Likewise,  Rjy  designs  are  considered  1  st 
order  designs.  Although  l^t  order  terms  in  an  Rjy  design  are  not  aliased  with  any  terms 
lower  than  3*^^  order,  the  2“^  order  terms  are  still  aliased  with  one  another.  Resolution 
five  designs  are  ideal  for  2*^^  order  models  since  the  l^f  order  terms  are  not  aliased  with 
any  terms  lower  than  4^^  order,  and  the  2>^^  order  terms  are  "free  of  confounding  with 
any  terms  lower  than  3^^  order.  Screening  designs  most  often  use  Rju  or  Rjv 
designs  to  determine  WHICH  of  many  variables  under  consideration  are  most  important. 
Designs  of  Ry  or  higher  are  generally  used  to  establish  HOW  the  input  variables  affect 
the  response(s).  Determining  WHY  the  process  behaves  as  it  does  entails  a  sequential 
process  of  scientific  inquiry  intended  to  establish  a  theoretical  basis  for  the  observed 
phenomena.  The  experimental  designs  used  to  address  the  fundamental  WHICH,  HOW, 
and  WHY  questions  provide  the  basis  for  empirical  and  mechanistic  model  building.(Box 
and  Draper,  1987;  11-14) 

The  designs  in  this  study  are  all  two  level  designs,  where  each  input  variable  is  at 
either  a  high  (+1)  or  low  (-1)  setting.  These  designs  are  used  to  estimate  models  with 
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terms  consisting  of  linear  combinr^'ons  of  the  input  variables  and  their  products. 
Appendix  C  provides  the  specific  designs  used  in  this  thesis. 

Analyzing  an  experiment  utilizes  the  RSM  techniques  of  multivariate  regression 
to  analyze  the  data  and  identify  the  one  best  model  (or  metamodel).  Several  statistical 
"goodness  of  fit"  measures  exist  to  assist  an  analyst  in  making  this  determination.  This 
information  is  frequently  presented  in  an  ANOVA  table  which  summarizes  the 
information  relating  the  least  squares  fit  of  the  model  to  the  data.  The  value 
represents  a  measure  of  how  much  of  the  structure  of  the  data  can  be  explained  by  the 
model.  An  R^  value  of  one  means  the  model  fits  the  data  perfectly.  As  the  number  of 
terms  in  the  model  increases,  the  r2  value  goes  up  (or  possibly  stays  the  same).  The 
adjusted  r2  value  takes  into  account  the  number  of  degrees  of  freedom  used  by  the 
model.  Each  term  in  the  model  uses  one  degree  of  jfreedom.  The  total  available  number 
of  degrees  of  fi’eedom  is  equal  to  the  number  of  data  points. 

The  Mean  Square  Error  (MSE)  is  an  estimate  of  the  variance  of  the  process,  if  the 
model  is  assumed  to  be  correct.  If  the  model  is  not  correct,  MSE  is  the  sum  of  the 
variance  due  to  the  process  and  the  lack  of  fit. 

The  partial  F  statistic  is  the  ratio  of  the  sum  of  squares  to  the  MSE  for  each  term 
in  a  specific  model.  It  represents  the  probability  that  the  coefficient  associated  with  each 
term  is  actually  zero. 

Mallow's  CP  statistic  requires  an  estimate  of  the  variance  of  the  process  under 
study.  If  the  number  of  terms  in  a  model  is  much  less  than  the  CP  statistic,  then  that 
model  does  not  mirror  the  process  under  study  well. 

Each  of  these  statistic^  measures  should  be  considered  when  analyzing  possible 
models.  No  single  measure  will  identify  the  "best"  model.  Simplicity  of  the  model  is 
also  an  important  feature.  A  simple  model  may  be  more  useful  than  a  more  complex 
model  that  fits  the  data  marginally  better.  For  example,  Newtonian  equations  of  motion 
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are  less  accurate  than  relativistic  equations  of  motion,  but  when  the  velocities  involved 
are  much  less  than  the  speed  of  light,  the  simpler  Newtonian  models  are  usually  used.  At 
these  velocities  the  differences  between  the  output  of  the  two  models  is  negligible. 

RSM  has  often  been  used  to  optimize  a  specific  simulation  output  as  a  fimction  of 
the  inputs  (Harvey,  Bauer,  and  Litko,  1993).  Analysts  use  simulations  to  study  complex 
situations  with  complex  input-to-output  responses.  Analysts  use  RSM  to  clarify  the 
relationship  between  the  simulations'  input  variables  and  the  simulations'  output. 

Once  the  relationship  between  the  inputs  and  outputs  is  estimated  as  a  metamodel, 
the  metamodel  is  optimized  using  a  variety  of  techniques  (Garrison,  1992).  If  the 
metamodel  is  a  linear  combination  of  variables,  the  magnitude  and  sign  of  the  coefficients 
associated  with  the  variables  indicate  the  contribution  each  variable  makes  to  the 
objective.  It  is  important  to  note  that  RSM  techniques  usually  scale  variables  to  +1  and  - 
1  values  (or  similar  set  of  scaled  values).  To  optimize  the  simulation  output,  the 
metamodel  estimated  using  the  scaled  variables  must  be  transformed  into  the  original 
variables.  Such  a  transformation  ^an  change  the  relative  values  of  the  coefficients. 

If  the  model  is  a  complex,  nonlinear  model,  then  steepest  ascent  search  techniques 
are  used  to  find  the  optimal  solutions.  Steepest  ascent  searches  are  commonly  used 
algorithms  which  attempt  to  locate  the  maximum  (or  minimum)  values  of  nonlinear 
functions.  A  commercial  package  known  as  GINO  was  used  to  optimize  the  complex 
metamodels.  GINO  supports  the  optimization  of  nonlinear  objective  functions  subject  to 
constraints. 

Conclusion 

TAG  THUNDER  is  a  complex  computer  simulation  of  a  theater-level  conflict.  In 
order  to  use  it  to  estimate  the  effectiveness  of  a  specific  set  of  aircraft,  the  analyst  must 
use  these  forces  optimally  relative  to  the  objectives  of  the  campaign.  RSM  can  be  used 
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to  identify  and  understand  the  relationships  that  exist  between  simulation  input  (aircraft 
apportionment)  and  simulation  output  (the  results  of  the  conflict).  Additional  sources  of 
information  on  the  techniques  and  methods  used  in  this  research  endeavor  are  listed  in  the 


bibliography.  Chapter  three  contains  more  information  on  the  specific  TAC  THUNDER 
experiments  conducted  as  part  of  this  thesis  and  the  resulting  metamodel. 
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III.  Baseline  Experiment 
IntroductiQn 


This  thesis  used  a  sequential  experimentation  approach  to  optimize  the  use  of 
aircraft  within  a  single  TAC  THUNDER  scenario.  Each  phase  in  the  investigation 
provided  information  and  insight  used  to  refine  and  direct  subsequent  phases  throughout 
the  research  effort.  The  general  iterative,  experimental  approach  consists  of  conjecture, 
design,  experimentation,  and  analysis.  This  chapter  presents  the  details  of  the 
conjecture,  design,  experimentation,  and  analysis  efforts  associated  with  the  baseline 
experiment. 

Conjecture 

Theater-level  campaign  analyses,  using  proper  analytical  treatment,  provide  the 
only  meaningful  way  of  estimating  the  contributions  of  systems  that  perform  disparate 
functions  in  wartime  (Goodson,  1993;1).  General  Goodson  defines  proper  analytical 
treatment  as  mathematical  search  algorithms  that  find  the  best  use  of  force.  Comparisons 
between  force  options  are  only  meaningful  if  all  the  force  options  have  been  used  to  their 
maximum  effectiveness.  Such  comparisons  are  difficult  because  theater-level  conflicts 
are  large  and  complex  by  nature,  and  the  relationships  between  force  employment  and 
battle  outcomes  are  not  known  ahead  of  time.  For  combat  models,  like  TAC  THUNDER, 
that  do  not  optimize  the  apportionment  of  forces,  an  analyst  must  input  the  best 
apportionment  for  each  resource  under  study  to  insure  that  the  outcomes  of  each  conflict 
reflect  the  capabilities  of  the  resources  available.  This  research  was  based  on  the  premise 
that  it  is  possible  to  estimate  the  relationship  between  the  air  apportionment  and  the 
outcomes  of  a  theater-level  scenario. 

The  objective  of  this  thesis  is  to  optimize  the  air  apportionment  with  respect  to 
each  of  three  measures  of  effectiveness  (MOE):  The  movement  of  the  forward  line  of 
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own  troops  (PLOT),  the  strength  of  friendly  ground  forces  when  the  opponent's  advance 
is  halted,  and  the  attrition  of  aircraft. 

This  research  used  an  unclassified  scenario  based  on  Desert  Storm.  The  original 
scenario  provided  by  CACI  had  to  be  modified.  The  original  scenario  modeled  the 
ground  war  phase  of  Desert  Storm.  That  scenario  was  too  unbalanced  to  show  a  strong 
relationship  between  the  best  use  of  aircraft  and  the  campaign  outcomes.  The  forces  in 
the  scenario  were  changed  to  reflect  a  strong  Iraqi  attack  from  Kuwait  into  Saudi  Arabia 
not  long  after  they  had  invaded  Kuwait.  The  coalition  ground  forces  were  oumumbered 
and  significant  additional  ground  forces  were  30  days  away.  The  objective  of  the 
coalition  commander  was  to  stop  the  Iraqi  advance  while  minimizing  the  loss  of  coalition 
air  and  ground  power.  It  was  presumed  that  once  the  PLOT  was  stopped,  this  objective 
had  been  met.  Additional  forces  would  arrive  later  and  eventually  attack,  but  the  best  use 
of  forces  in  those  situations  is  not  likely  to  be  the  same  as  during  this  initial  advance.  The 
optimal  air  apportionment  sought  related  only  to  halting  the  opponent's  advance.  The 
optimal  apportionment  for  subsequent  operations  was  not  part  of  this  research  effort. 
Additional  details  of  this  scenario  are  available  in  Appendix  A. 

Design 

There  are  12  types  of  aircraft  and  17  missions  defined  for  this  scenario.  A  30  day 
war  would  require  12x17x30  or  6120  variables  to  represent  each  aircraft/mission/day 
combination.  If  the  relationship  between  the  inputs  and  the  outputs  were  known  and 
linear,  a  linear  program  could  optimize  the  air  apportionment.  Unfortunately,  these 
relationships  are  not  known,  so  some  other  approach  is  required. 

RSM  was  chosen  as  the  best  approach,  given  the  nature  of  the  problem,  but  it 
required  reducing  the  number  of  variables  to  a  manageable  level.  The  number  of 
variables  impacts  the  number  of  simulation  runs  required  to  estimate  the  relationship 
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between  the  air  apportionment  and  battle  outcomes.  Simulating  ten  interations  of  a  30 
day  conflict  took  about  six  hours  on  a  Sun  workstation.  The  computational  time  required 
to  do  hundreds  of  runs  exceeds  the  time  available  for  many  study  efforts  including  this 
thesis  effort.  Studies  accomplished  after  a  decision  is  made  tend  to  be  of  little  value  to  a 
decision  maker. 

Therefore,  the  following  assumptions  were  made  to  simplify  the  problem: 

-  The  opponent's  air  apportionment  was  fixed.  It  represented  our  best  estimate  of  his  air 
plan.  The  simulation  results  applied  only  to  this  particular  strategy. 

-  Only  the  "best"  aircraft  for  each  mission  were  included  in  this  simulation:  F-15's  for  air 
superiority  missions,  F-15E's  for  INT  and  BAI,  FI  1  Ts  for  OCA,  A-lO's  for  CAS.  It  is 
assumed  that  if  other  aircraft  were  assigned  to  these  missions,  the  results  would  not  be 
better  (they  may  be  the  same  or  worse). 

-  A  constant  air  apportionment  is  acceptable  for  the  first  30  days  of  this  scenario  since  the 
situation  remains  relatively  constant. 

The  input  variables  were  the  number  of  aircraft  assigned  to  each  of  the  missions 
listed  on  pages  2-2  through  2-4.  Not  all  of  these  missions  necessarily  contributed 
significantly  in  this  scenario.  Therefore,  a  screening  design  was  chosen  to  identify  the 
missions  that  were  most  significant.  A  second  experiment  could  be  run  later  using  only 
these  signi  leant  variables  to  develop  a  higher  order  model. 

A  fi-actional  factorial  design  was  chosen  for  the  initial  experiment.  It 
would  permit  estimation  of  the  main  effects  and  could  be  supplemented  with  an 
additional  fraction,  if  necessary.  The  2^’5  indicates  the  experiment  consists  of  16  (2^) 
design  points,  and  the  "III"  indicates  that  it  is  a  resolution  three  design.  Statgraphics 
software  was  used  to  help  design  the  experiment  and  analyze  the  data.  Statgraphics 
generated  a  design  matrix  that  showed  the  value  of  each  input  variable  for  each  of  the  16 
runs.  These  values  were  coded  as  +1  or  - 1 .  Each  mission  had  a  maximum  and  minimum 
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number  of  aircraft  that  were  assigned  to  fly  that  mission  on  the  first  day  of  the  war. 

These  values  define  the  boimdaries  of  the  input  variables.  The  estimated  response  surface 
was  confined  to  points  bounded  by  these  minimum  and  maximum  values. 

Experimentation 

This  section  describes  the  implementation  of  the  fictional  factorial  design. 
Important  decisions  on  the  implementation  had  to  be  made.  These  decisions  included 
how  many  times  TAC  THUNDER  would  be  run  at  a  single  design  point,  and  what  should 
be  the  maximum  and  minimum  number  of  aircraft  assigned  to  each  mission. 

The  output  of  a  single  TAC  THUNDER  run  is  a  random  variable.  Additional  runs 
with  identical  inputs  allow  a  more  precise  estimation  of  the  mean  response  of  TAC 
THUNDER  to  those  inputs.  Additional  runs  also  require  more  time  and  computer 
resources.  For  this  experiment,  it  was  decided  to  repeat  each  design  point  ten  times.  Ten 
repetitions  reduced  the  uncertainty  bounds  on  mean  result  at  each  data  point.  Additional 
runs  did  not  improve  the  uncertainty  bounds  enough  to  justify  allocating  the  additional 
resources. 

The  maximum  and  minimum  number  of  aircraft  assigned  to  each  mission  defined 
the  variable  space  for  this  experiment.  The  lower  bound  for  each  mission  class  was 
selected  to  be  the  fewest  number  of  aircraft  a  commander  might  realistically  assign.  For 
example,  a  commander  would  never  voluntarily  have  no  defensive  air  capability.  The 
maximum  value  was  selected  as  the  largest  number  of  aircraft  a  commander  might  assign 
and  still  not  encounter  significant  diminishing  returns.  Clearly,  if  enough  aircraft  are 
currently  assigned  to  destroy  every  interdiction  target  in  the  theater,  the  value  of 
additional  interdiction  sorties  is  very  small.  Appendix  A  contains  the  maximum  and 
minimum  values  of  aircraft  assigned  to  each  mission  class. 
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The  TAC  THUNDER  files  that  contain  information  on  each  squadron  were 
changed  to  reflect  the  appropriate  number  of  each  type  of  aircraft  for  each  design  point. 
The  TAC  THUNDER  air  apportionment  files,  however,  assign  a  percentage  of  each 
mission  class  to  specific  missions  rather  than  specific  numbers.  In  the  case  of  CAS  and 
BAI,  this  problem  was  solved  by  assigning  100%  of  the  aircraft  of  a  specific  class 
(ground  support  and  multi-role)  to  those  missions.  The  multi-role  class  was  changed  to 
perform  as  if  they  were  all  F15E's.  The  available  deep  strike  aircraft  were  divided 
between  OCA  and  INT  missions.  The  air  superiority  class  was  divided  between  DCA, 
FSWP,  and  AIRESC  missions.  The  percentages  of  each  mission  class  assigned  in  the  air 
apportionment  file  were  adjusted  so  that  the  number  initially  assigned  to  each  mission 
reflected  the  high  or  low  values  associated  with  each  mission  for  that  run.  For  example, 
if  DCA,  FSWP,  and  AIRESC  were  all  at  their  max  or  min  values  (72  or  12)  then  the  air 
superiority  mission  class  would  be  assigned  33%  to  DCA  and  AIRESC  missions  and 
34%  to  the  FSWP  mission.  If  the  design  point  required  72  FSWP  and  12  DCA  and 
AIRESC  missions,  the  air  superiority  mission  class  would  be  divided  as  follows:  75%  to 
FSWP,  and  12.5  %  to  DCA  and  AIRESC  respectively. 

TAC  THUNDER  has  a  graphics  post  processor  that  creates  user-defined  macros 
to  display  the  results  of  each  run.  Due  to  the  tremendous  amount  of  data  associated  with 
each  run,  a  script  was  created  that  stored  the  information  for  the  macros  and  then  deleted 
everything  else  before  starting  the  next  repetition  of  that  run.  This  was  a  compromise 
between  the  desire  for  large  statistical  samples  and  the  limitations  on  time.  Each  iteration 
took  approximately  30  minutes  to  run  on  a  Sparc  II  workstation.  Three  separate  runs 
were  done  simultaneously  on  one  machine.  Approximately  18  hours  later,  those  runs 
were  finished  and  new  set  of  runs  was  started.  Occasionally  the  TAC  THUNDER 
program  would  generate  an  error  that  would  stop  a  batch  run  and  require  restarting  the 
run. 
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The  output  for  each  set  of  runs  was  condensed  into  one  file  and  printed  out  four  to 
a  page,  double  sided.  It  was  necessary  to  look  at  the  PLOT  movement  to  determine 
which  day  the  PLOT  movement  reached  its  final  position.  The  air  and  ground  attrition 
numbers  used  were  those  associated  with  that  day.  The  10  iterations  of  runs  were 
averaged  to  yield  the  response  inputs  for  statgraphics.  The  design  and  the  results  of  the 
initial  screening  experiment  are  listed  in  Appendix  C. 

Analysis. 

Estimating  the  Response  Surfaces:  Initially,  the  aircraft  losses  were  expressed 
as  a  percentage  of  the  initial  inventory  available.  This  response  was  unsuitable  because 
the  variance  of  the  response  was  a  function  of  the  value  of  the  response.  One  of  the 
fundamental  assumptions  of  regression  is  that  the  variance  of  the  response  is  constant 
throughout  the  design  region.  Expressing  the  losses  in  terms  of  aircraft  lost  essentially 
eliminated  the  problem  of  heteroschodasticity. 

The  original  objective  of  the  screening  design  was  to  identify  WHICH  of  the 
variables  were  important.  However,  high  quality  metamodels  were  estimated  with  the 
data  fi'om  the  screening  design  experiment,  rendering  subsequent  higher  resolution 
designs  unnecessary. 

The  MOEs  were  strongly  correlated  to  the  experimental  setting  of  the  CAS  and 
INT  missions.  In  Pigures  3.1  through  3.3,  a  lowercase  "c"  or  "i"  indicates  a  low  level  for 
CAS  or  INT,  while  a  capital  "C"  or  "I"  indicates  a  high  level.  Larger  PLOT  values 
represent  a  deeper  Iraqi  penetration  into  Saudi  Arabia. 
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Figure  3.1  FLOT  results  grouped  by  CAS  and  INT 

Notice  the  overwhelming  effect  of  CAS  missions  in  Figure  3.1.  The  higher 
proportion  of  CAS  missions  significantly  reduced  the  Iraqi  advance.  INT  missions  also 
contributed  to  significantly  reducing  the  FLOT  movement. 


Figure  3.2  Ground  Strength  results  grouped  by  CAS  and  INT 

In  Figmre  3.2,  CAS  and  INT  missions  significantly  influence  the  response  of  the 
MOE  (ground  strength).  Higher  levels  of  CAS  and  INT  generally  led  to  higher  levels  of 
remaining  allied  ground  strength. 
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Figure  3.3  A/C  attrition  grouped  by  CAS  and  INT 
Figure  3.3  shows  one  of  the  most  unexpected  results  of  this  study.  The  fewest 
number  of  aircraft  tended  to  be  lost  when  CAS  missions  were  at  the  high  level.  A  closer 
examination  of  the  data  revealed  that  the  high  level  of  CAS  generally  resulted  in 
objectives  being  met  more  expeditiously.  The  shorter  conflict  reduced  the  total  number 
of  aircraft  sorties.  Fewer  sorties  exposed  to  the  threat  resulted  in  fewer  aircraft  lost. 
Another  possible  explanation  is  that  the  CAS  missions  reduced  the  intrinsic  air  defenses 
of  the  Iraqi  units.  These  intrinsic  air  defenses  consist  of  portable  surface-to-air  missiles 
(SAM's),  mobile  SAM's,  and  gxm  systems  used  in  air  defense. 

The  rest  of  this  chapter  presents  the  metamodels  estimated  from  the  screening 
design  data.  The  ANOVA  table  associated  with  metamodels  as  well  as  diagnostic  plots 
are  provided  to  assist  the  reader  in  understanding  how  the  various  metamodels  were 
estimated.  The  estimation  of  the  FLOT  model  includes  a  description  of  how  examination 
of  the  residuals  led  to  the  inclusion  of  a  nonlinear  term  in  the  metamodel. 
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The  ANOVA  for  the  purely  linear  PLOT  model  is  given  in  Table  3.1. 


Input  Variablg 

CAS 

INT 

Total  Error 
Total  Corrected 


Table  3.1  ANOVA  of  PLOT  model  (Linear) 

Sum  of  Squares _ DE _ MsaiLSa. _ EiBatifi P-value 

2594274  1  2594274  309  .0000 

179289  1  179289  21.3  .0000 

109194  13  8400 

2882757  15 


r2=  0.962  r2  (adjusted  for  degrees  of  freedom)  =  0.956 


While  the  adjusted  R^  of  the  linear  PLOT  model  was  high,  the  distinct  U-shape  of  the 
residual  plot  in  Pigure  3.4  indicates  the  need  for  a  higher  order  term  in  the  model. 

Ideally,  the  residuals  reflect  random  deviations  from  the  model  that  are  only  "noise” 
terms  and  are  distributed  normally  about  a  mean  of  zero.  A  distinct  pattern  in  the  plot  of 
the  residuals,  such  as  is  foimd  in  Pigure  3.4,  indicates  the  presence  of  an  unaccoimted 
effect.  Adding  the  CASxINT  interaction  term  to  the  model  removed  all  discernible 
patterns  from  the  residual  plot. 
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Diagnostic  Piot  for  PLOT 


-100  100  300  500  700  900  1100 

predicted 

Figure  3.4  Plot  of  residuals  for  PLOT 
The  final  estimated  PLOT  model  is: 

FLOT=  1061  -  4.460*  C4S  -  3.852  *  INT  +  0.01525*  C45  *  INT  (Eq  3-1) 

Where: 

PLOT  is  the  total  advance  of  the  enemy  in  kilometers. 

CAS  is  the  number  of  close  air  support  sorties  assigned. 

INT  is  the  number  of  interdiction  sorties  assigned. 
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The  ANOVA  for  the  PLOT  model  is  given  in  TABLE  3.2. 


Table  3.2  ANOVA  for  the  PLOT  model  (nonlinear) 


CAS 

2594274 

1 

2594274 

3362 

.0000 

INT 

179289 

1 

179289 

232 

.0000 

CAS*INT 

99935 

1 

99935 

130 

.0000 

Total  Error 

9259 

12 

772 

Total  Corrected 

2882757 

15 

r2=  0.9968  r2  (adjusted  for  degrees  of  freedom)  =  0.9960 


The  adjusted  r2  of  .99  represents  a  nearly  perfect  fit.  The  F-test  of  the  CASxINT 
term  is  highly  significant,  and  the  new  equation  models  the  response  almost  perfectly. 
Equation  3-1  matches  the  data  representing  the  PLOT  movement  nearly  exactly. 

The  estimated  model  for  the  ground  strength  remaining  at  the  end  of  the  first 
phase  of  the  campaign  is: 


GrdStr  =  45. 847  +  0.080444  *  CAS  +  0.063021  *  INT  (Eq  3-2) 

where 

Grd  Str  is  the  combat  firepower  of  the  fiiendly  ground  forces  when  the  PLOT  is 

halted. 


Table  3.3  gives  the  ANOVA  for  the  ground  strength  model  defined  in  Equation  3-2. 


Table  3.3  ANOVA  for  ground  strength  model 


Input  Variable _ Sum  of  Squares 

CAS  1207.6 

INT  146.4 

Total  Error  34.0 

Total  Corrected  1388 


J2E _ Mean  Sq. _ F-Ratio  P-value 

1  1207.6  462  .0000 

1  146.4  56  .0000 

13  2.6127 

15 


r2=  0.9755  r2  (adjusted  for  degrees  of  freedom)  =  0.971 8 
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An  adjusted  of  .97  represents  an  excellent  fit.  The  residuals  (which  represent 
the  distribution  of  the  error  term)  appear  to  be  distributed  normally  based  on  the  residual 
plot  and  the  normal  probability  plot  of  the  residuals. 

The  estimated  model  for  the  aircrafi  attrition  is: 

AirAtr  =  7.66+.160  *  OCA  +  0.0152  *  AIRESC  +  0.0477  *  BARCAP 
+0.1 102  *  FSWP  -  0.0363  *  CAS  +  0.0475  *  BAI 

-0.00\Q46*  OCA*  INT  + 0.1206*  INT- 0.00199*  AIRESC  *FSWP  ^  ^ 
-O.OOeSl* CAS  *INT 

Where 

AirAtr  is  the  number  of  friendly  aircrafi  lost. 

OCA  represents  the  number  of  OCA  missions  flown  on  the  first  day  of  the  war. 
AIRESC  represents  the  number  of  AIRESC  missions  flown  on  the  first  day  of  the 
war. 

BARCAP  represents  the  number  of  BARCAP  missions  flown  on  the  first  day  of 
the  war. 

FSWP  represents  the  number  of  FSWP  missions  flown  on  the  first  day  of  the  war. 
5/47  represents  the  number  of  BAI  missions  flown  on  the  first  day  of  the  war. 

Except  for  attrition,  the  input  variables  are  constant. 
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Table  3.4  gives  the  ANOVA  for  the  air  attrition  model  defined  in  Equation  3-3. 

Table  3.4  ANOVA  for  the  air  attrition  model 


OCA 

141.0 

1 

141.0 

161 

.0001 

AIRESC 

139.8 

1 

139.8 

159 

.0001 

BARCAP 

32.8 

1 

32.8 

37 

.0017 

FSWP 

10.1 

1 

10.1 

11.5 

.0195 

CAS 

890 

1 

890 

1014 

.0000 

INT 

113 

1 

113 

129 

.0001 

BAI 

83.3 

1 

83.3 

95 

.0002 

OCA*INT 

29.4 

1 

29.4 

34 

.0022 

AIRESC*FSWP 

51.5 

1 

51.5 

59 

.0006 

CAS*INT 

200 

1 

200 

227 

.0000 

Total  Error 

4.39 

5 

.87762 

Total  Corrected 

1694.2 

15 

r2=  0.9974 1  r2  (adjusted  for  degrees  of  freedom)  =  0.99223 


Table  3.4  shows  the  fit  for  an  1 1  term  model  of  the  air  attrition.  The  model  for  air 
attrition  is  an  excellent  fit  to  the  data  and  each  of  the  terms  appears  to  be  significant 
according  to  the  partial  F-test.  Obviously  air  attrition  is  a  complex  process.  The  only 
concern  with  this  model  is  the  low  degrees  of  freedom  for  the  error  term. 

Optimization  of  the  Response  Surfaces:  GINO,  a  steepest  gradient  search  tool, 
was  used  to  optimize  the  air  attrition  model.  The  other  two  models  were  of  sufficient 
simplicity  to  be  optimized  analytically. 


The  PLOT  model  (Equation  3-1)  was  optimized  by  taking  partial  derivatives  with 
respect  to  each  of  the  variables  and  equating  them  to  zero.  The  partial  derivatives  of  the 
PLOT  model  are: 


=  -4.46+.  0 1 525  *  /AT 

dCAS 

dFLOT  ^  _3  352+.oi525*C45 

dim 


(Eq3-4) 


The  minimum  estimated  PLOT  movement  occurs  for  253  CAS  aircraft  and  292 
INT  aircraft.  Both  of  these  values  are  outside  the  design  region.  The  minimum  estimated 
PLOT  movement,  therefore,  occurs  on  the  boundary  of  the  design  region.  In  this  case, 
the  minimum  occurs  for  216  CAS  aircraft  and  96  INT  aircraft. 

The  grovmd  strength  metamodel  (Equation  3-2)  is  a  linear  combination  of  CAS 
and  INT  missions.  The  ground  strength  is  maximized  when  CAS  aircraft  and  INT 
aircraft  are  maximized.  Once  again,  the  optimal  result  occurs  for  216  CAS  aircraft  and 
96  INT  aircraft. 

Complex  nonlinear  models  such  as  the  air  attrition  model  are  not  as  easy  to 
analyze.  GINO,  a  commercitdly  available  analysis  software,  was  used  to  optimize  the  air 
attrition  equation  with  constraints.  The  optimization  problem  was  set  up  as  follows; 


Minimize:  7.664  +  0.1602*OCA  +  0.01519*AIRESC  +  0.0477*BARCAP  + 

0.1102*PSWP  -  0.03473*CAS  +  0.04752*BAI  -  0.001 046*OCA*INT  + 
0.1206*INT-  0.001 993  *AIRESC*PSWP  -  0.00065 1*CAS*INT; 

Subject  to  the  following  constraints: 

OCA  +  AIRESC  +  BARCAP  +  PSWP  +  CAS  +  INT  <  xxx  ( xxx  represents  the 
maximum  number  of  aircraft  in  the  theater) 
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.076885*CAS  +  .063021*  INT  +  45.8  >  yyy  (yyy  represents  the  minimum 
acceptable  ground  strength  level.  If  yyy  =  0  then  this  constraint  is  inactive.) 
AIRESC  -  OCA  -  4*INT  <  0  (This  constraint  limits  the  number  of  escort 
missions  to  the  maximum  possible  based  on  the  number  of  escortable  missions 
plaimed.) 

OCA  >18  (The  minimum  number  of  OCA  missions) 

AIRESC  >12  (The  minimum  number  of  AIRESC  missions) 

FSWP  >  12  (The  minimum  number  of  FSWP  missions) 

BARCAP  >12  (The  minimum  number  of  BARCAP  missions) 

BAI  >  0  (The  minimum  number  of  BAI  missions) 

INT  >  0  (The  minimum  number  of  INT  missions) 

CAS  >  0  (The  minimum  number  of  CAS  missions) 

OCA  <  96  (The  maximum  number  of  OCA  missions) 

AIRESC  <  72  (The  maximum  number  of  AIRESC  missions) 

FSWP  <  72  (The  maximum  number  of  FSWP  missions) 

BARCAP  <  72  (The  maximum  number  of  BARCAP  missions) 

BAI  <  98  (The  maximum  number  of  BAI  missions) 

INT  <  96  (The  maximum  number  of  INT  missions) 

CAS  <216  (The  maximum  number  of  CAS  missions) 

The  GINO  problem  was  initally  executed  with  the  maximum  number  of  aircraft 
within  the  theater  equal  to  56.  Setting  all  the  variables  at  the  lowest  level  results  in  a  total 
of  56  aircraft  in  the  theater.  The  apportionment  was  recorded  and  then  number  of  aircraft 
in  the  theater  v/as  increased  by  20.  Again,  the  problem  was  solved  and  the  apportionment 
was  recorded.  Several  starting  values  were  used  to  avoid  aircraft  apportionment  solutions 
that  were  only  local  optima.  The  process  continued,  adding  20  aircraft  to  tlie  theater  at  a 
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time,  until  the  maximum  number  of  aircraft  in  the  theater  was  reached.  The  optimal 
apportionment  allocated  the  next  216  aircraft  to  CAS  missions.  Once  the  number  of 
aircraft  in  the  theater  exceeded  272  (216+56),  available  aircraft  were  assigned  to  INT 
missions.  Finally,  once  the  number  of  aircraft  in  the  theater  exceeded  368  (272+96),  the 
AIRESC  and  FSWP  missions  were  maximized.  No  other  missions  contributed  to 
reducing  the  aircraft  attrition. 

Note  that  the  same  apportionment  of  216  CAS,  96  INT,  72  AIRESC,  72  FSWP, 

12  BARCAP,  8  SIAM,  ?»nd  18  OCA  aircraft  optimizes  all  three  MOEs!  For  this  scenario, 
there  was  no  tradeoff  between  air  losses  and  ground  losses  or  FLOT  movement!  The 
optimal  apportionment  begins  with  CAS,  then  INT,  and  finally  a  mixture  of  AIRESC  and 
FSWP  missions.  CAS  missions  are  the  most  effective  mission  for  all  three  MOEs.  INT 
missions  are  also  effective  in  all  three  MOEs.  The  FSWP  and  AIRESC  missions  reduce 
the  aircraft  attrition  while  not  having  any  significant  effect  on  the  other  two  MOEs. 
Although  not  specifically  included  in  all  the  models,  BARCAP,  FSWP,  AIRESC,  SJAM, 
and  OCA  have  positive  minimum  values.  These  terms  would  not  appear  as  significant  in 
the  model  if  the  minimum  levels  satisfied  the  requirements  levied  by  the  associated 
MOE's.  SJAM  and  SSUP  missions  do  not  appear  to  have  a  significant  impact  on  any  of 
the  MOE's  examined  in  this  study. 

Figures  3.5,  3.6,  and  3.7  show  how  the  MOEs  vary  as  the  number  of  aircraft  in  the 
theater  increases.  These  figures  are  based  on  the  optimal  allocation  of  aircraft  given  the 
number  of  aircraft  in  the  theater.  The  solid  squares  on  the  charts  indicate  the  average 
predicted  response  based  on  the  number  of  aircraft  in  the  theater.  The  hollow  squares 
indicate  a  95%  confidence  limit  value.  The  95%  confidence  limit  values  were 
calculated  using  Equation  3-5  with  alpha  equal  to  0. 1 .  Equation  3-5  is  the  equation  for  a 
two  sided  confidence  limit  of  90%.  Since  the  normal  function  is  symmetric,  choosing  the 
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worst  case  boundary  yields  a  95%  boundary.  Since  the  experimental  design  was 
orthogonal,  the  covariance  matrix  is  an  identity  matrix  and  Eqitation  3-5  reduces  to  3-6. 


Y±t(n-p-l,l-^a)*  Syp^  (Eq  3-5) 

Where: 

Xq  is  a  vector  representing  the  apportionment  of  aircraft 

r±t(«-p-l,l-^a)'sJXX  (Eq3-6) 

Figure  3.5  reveals  that  the  metomodel  predicts  a  negative  number  of  aircraft  lost 
when  there  are  more  than  400  aircraft  in  theater.  Clearly,  negative  numbers  of  aircraft 
lost  makes  no  sense.  A  reasonable  interpretation  of  the  chart  is  that  the  probability  of 
losing  more  than  5  aircraft  (out  of  400+)  is  less  than  5%.  The  model  does  not  accurately 
estimate  the  mean  number  of  aircraft  lost  at  this  extreme  portion  of  the  response  surface. 

Air  Attrition 


20  _ 


Figure  3.5  Aircraft  attrition  as  a  function  of  the  number  of  aircraft  in  the  theater 
based  on  the  optimal  apportionment  of  aircraft 
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Ground  Strength 


#  of  A/C  in  Theater 


Figure  3.6  Remaining  ground  strength  as  a  function  of  the  number  of  aircraft  in  the 

theater 

PLOT 


#  of  A/C  in  Theater 


Figure  3.7  The  total  FLOT  movement  as  a  function  of  the  number  of  aircraft  in  the 

theater 

Figures  3.5  through  3.7  provide  an  analyst  an  upper  bound  on  the  MOEs  as  a 
function  of  aircraft  in  the  theater  assuming  an  optimal  air  apportionment.  No  non- 
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material  solution  exists  that  will  satisfy  a  requirement  that  exceeds  the  optimum  MOE 
responses  depicted  in  Figures  3.5  through  3.7. 

Conclusion 

It  is  important  to  emphasize  that  the  results  of  this  research  cannot  be  generalized 
beyond  the  assumptions  and  limitations  of  the  scenario  examined.  CAS  missions  seem  to 
be  very  effective  in  reducing  PLOT  movement,  reducing  the  attrition  of  front  line  troops, 
and  reducing  the  number  of  aircraft  lost.  The  first  two  results  are  expected  since  CAS 
missions  attack  opposing  units  on  the  front  lines.  The  third  result  was  unexpected.  Is  it 
due  to  the  nature  of  this  scenario  or  is  it  an  anomaly  of  the  TAC  THUNDER  model? 
Figures  3.3  through  3.5  show  the  predicted  response  based  on  the  optimal  allocation  of 
aircraft.  For  this  scenario,  optimality  is  achieved  by  allocating  aircraft  first  to  CAS,  then 
to  INT,  and  finally  to  a  even  mix  of  FSWP  and  AIRESC.  Exact  values  are  calculated 
using  GINO  to  optimize  the  allocation  for  a  given  number  of  aircraft  assigned  to  the 
theater. 

It  should  be  noted  that  the  minimum  number  of  aircraft  assigned  to  the  OCA, 
BARCAP,  FSWP,  and  AIRESC  missions  was  set  greater  than  zero.  The  absence  of  OCA 
and  BARCAP  from  the  optimal  apportionment  implies  that  assigning  additional  aircraft 
to  these  missions  did  not  improve  the  MOEs  examined  in  this  study.  It  is  unclear 
whether  the  minimum  number  of  OCA  and  BARCAP  aircraft  was  sufficient  to  perform  a 
vital  mission,  or  if  these  two  missions  did  not  contribute  to  accomplishing  the  overall 
objectives. 

It  is  clear  that  RSM  can  estimate  the  response  of  TAC  THUNDER  to  various  air 
apportionments.  Once  the  response  surface  has  been  estimated,  the  apportionment  can  be 
optimized  and  tradeoffs  identified.  The  response  surface  also  provides  the  analyst  with 
insight  into  the  performance  of  TAC  THUNDER. 
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IV.  Sensitivity  Analysis 


Introduction 

The  optimal  solution  foimd  in  chapter  three  is  based  on  a  fixed  scenario.  Many  of 
the  features  of  that  scenario  are  not  known  exactly.  A  commander  might  well  ask,  "How 
sensitive  is  the  solution  to  changes  in  this  fixed  scenario?"  Sensitivity  analysis  can  be 
used  to  investigate  the  effect  of  changes  in  the  scenario  on  the  "optimal"  air 
apportionment. 

The  capabilities  of  men  and  machines  must  be  quantified  and  input  into  combat 
simulations.  Information  on  one's  capabilities  is  usually  more  available  than  that  of  an 
adversary.  Therefore,  the  estimates  of  an  opponent's  capabilities  are  less  certain  than  the 
estimates  of  our  own  capabilities.  The  uncertainty  about  capabilities  becomes  even  more 
pronounced  when  one  is  estimating  the  future  capabilities  of  armed  forces.  The  sensitivity 
analysis  in  this  chapter  focuses  on  two  key  parameters  of  the  opponent's  capabilities:  their 
Air-to-Air  Probability  of  Kill  (AAPk)  and  their  Surface-to-Air  Probability  of  Kill 
(SAPk). 

These  values  represent  the  probability  that  an  enemy's  attack  will  destroy  an 
aircrafi,  given  that  it  has  been  detected  and  attacked.  Increasing  these  values  represents 
an  opponent  with  improved  quality  or  utilization  of  its  systems.  In  designing  the 
experiment  for  this  analysis,  the  high  level  for  both  the  AAPk  and  SAPk  values  were 
established  as  twice  their  original  values.  This  enhanced  performance  increased  the 
enemy's  Pk  values  to  a  level  comparable  with  our  own  capabilities.  The  low  level  used 
was  the  same  as  in  chapter  three.  Another  candidate  for  sensitivity  analysis  was  the  set  of 
Air-to-Surface  Pk  (ASPk)  values.  However,  the  ASPk  values  were  already  at  parity  with 
our  own  systems,  and  increasing  them  would  not  have  been  militarily  realistic. 

Therefore,  the  Air-to-Surface  Pk  values  are  not  part  of  the  sensitivity  analysis. 
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A  second  experiment  was  conducted,  adding  the  AAPk  and  SAPk  values  to  the 
seven  significant  factors  found  in  the  original  response  surface.  New  response  surfaces 
were  fit  to  estimate  the  relationship  between  the  AAPk,  SAPk,  and  an  MOE.  A  worst 
case  response  surface  was  also  developed  for  each  MOE.  The  worst  case  response 
surfaces  were  optimized  for  each  MOE  using  GINO. 

These  response  siufaces  indicate  how  TAG  THUNDER  responds  to  various 
inputs  and  provides  useful  information  for  imderstanding  TAG  THUNDER.  Several  non- 
intuitive  effects  were  found  which  indicate  the  possibility  that  the  TAG  THUNDER 
algorithms  may  not  be  operating  in  a  manner  consistent  with  our  understanding  of 
combat. 

Operational  Uncertainty  Analysis 

A  design  was  used  for  this  sensitivity  analysis.  Ten  repetitions  were 

again  conducted  at  each  of  the  32  design  points.  A  resolution  four  design  is  one  in  which 
the  linear  terms  associated  with  each  variable  are  confounded  only  with  third  and  higher 
order  terms.  Some  second  order  terms  are  confounded  with  other  second  order  terms. 

Appendix  B  describes  how  a  simulated  annealing  program  was  developed  to 
optimize  the  assignment  of  input  variables  to  design  factors  A  through  I.  A  ten  variable 
problem  has  approximately  10  factorial  (approximately  4  million)  combinations  of 
possible  assignments.  This  program  attempts  to  find  the  assignment  which  minimizes  the 
confounding  of  two  factor  interactions  thought  to  be  significant.  Using  this  technique 
permited  estimation  of  the  selected  two  factor  interactions  with  a  resolution  four 
experimental  design  in  lieu  of  a  more  time  consuming  resolution  5  design.  Appendix  G 
contains  the  data  used  in  this  chapter. 
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Response  surfaces  were  developed  for  each  MOE  using  least  squares  regression. 
The  model  for  the  PLOT  is: 

PLOT  =  979+(-2.050C4+2.09FS»T+0.0435BARC  AP>44P)fc-2.52C45- 
6 1  ASAPkA.9INTM)M26FSWP*BAI-m92im*OCA- 
O.Q50BAI*OCA  (Eq4-1) 

Where 

AAPk  is  the  level  (-1  to  +1)  of  the  Air-to-Air  Pk  value. 

SAPk  is  the  level  (-1  to  +1)  of  the  Surface-to-Air  Pk  value. 

OCA  is  the  number  of  OCA  missions  assigned  to  aircraft  on  the  first  day. 

FSWP  is  the  number  of  FSWP  missions  assigned  to  aircraft  on  the  first  day. 

CAS  is  the  number  of  CAS  missions  assigned  to  aircraft  on  the  first  day. 

INTis  the  number  of  INT  missions  assigned  to  aircraft  on  the  first  day. 

BAl  is  the  number  of  BAI  missions  assigned  to  aircraft  on  the  first  day. 

BARCAP  is  the  number  of  BARCAP  missions  assigned  to  aircraft  on  the  first  day. 

Negative  one  represents  a  Pk  value  equal  to  the  baseline  estimate.  A  positive  one 
represents  a  Pk  value  equal  to  twice  the  baseline  case.  The  ANOVA  for  the  PLOT  model 

in  Equation  4-1  is  given  in  Table  4. 1 . 
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Table  4.1  ANOVA  table  for  PLOT  (sensitivity  model) 


Input  Variable 


CAS 

INT 

SAPk 

INT*OCA 

BAI*OCA 

AAPk*OCA 

AAPk*FSWP 

AAPk*BARCAP 

FSWP*BAI 


Total  Error 
Total  Corrected 


Sum  fl£.Sauarss 
4137989 
695256 
120786 
124052 
136477 
98014 
125325 
125851 
120565 
536071 
659161 


fi» 

1 

4137989 

146 

.0000 

1 

695256 

26.3 

.0000 

1 

120786 

4.73 

.0439 

1 

124052 

4.69 

.0414 

1 

136477 

5.16 

.0332 

1 

98014 

3.71 

.0672 

1 

125325 

4.76 

.0405 

1 

125851 

4.76 

.0405 

1 

120565 

4.56 

.0441 

21 

84135 

31 


r2=  0.90 


r2  (adjusted  for  degrees  of  freedom)  =  0.87 


Figure  4.1  shows  how  the  blocking  the  observed  FLOT  movement  by  CAS  and 
INT  levels  graphically  depicts  the  impact  these  two  variables  have  on  the  response.  In 
Figure  4.1,  a  capital  "C"  or  "I"  indicates  a  high  level  of  the  CAS  or  INT  variable.  A 


lowercase  "c"  or  "i"  indicates  a  low  level  of  the  CAS  or  INT  variable. 


Figure  4.1  FLOT  grouped  by  CAS  and  INT 

Blocking  the  data  can  often  yield  better  models  Unfortunately,  grouping  the  data 
this  way  does  not  really  help  to  optimize  the  apportionment  unless  the  optimization  is 
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restricted  to  a  situation  where  the  CAS  or  INT  values  are  permanently  set  at  their  +1  or  -1 
values. 

Figure  4.1  shows  three  uncharacteristic  responses.  If  CAS  or  INT  was  the  only 
variable  at  the  high  level,  then  a  large  PLOT  response  could  still  occur.  Likewise, 
setting  the  other  missions  to  their  high  level,  except  for  CAS  and  INT,  was  sufficient  to 
stop  the  Iraqis  quickly.  Figure  4.1  is  useful  in  visually  presenting  the  impact  that  CAS 
an  "  missions  have  on  the  PLOT  MOE. 

Similarly,  the  ground  strength  response  surface  was  estimated  as: 

GrdStr  =  85.53  +{Q.Qmi51AIRESC+(i.\91SAPk-0.00nBARCAP)AAPk- 
0.0074F’5IFP+0.0064C45+0.019565^;?C4P-0.000489A47- 
0.005 1 1  OCA+Q.mQ2A%BAI*OCA-^ 

SAPki-Qmi2FSWP-Qm  1 54C^5-0.00306/A^  (Eq  4-2) 

Where; 

Grd  Str  is  the  combat  firepower  of  friendly  ground  units  when  phase  one  is 
completed. 

BARCAP  is  the  number  of  BARCAP  missions  assigned  to  aircraft  on  the  first  day. 
AIRESC  is  the  number  of  AIRESC  missions  assigned  to  aircraft  on  the  first  day. 
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Table  4.2  gives  the  ANOVA  associated  with  the  ground  strength  model  associated 


with  Equation  4-2. 

Table  4.2  ANOVA  for  ground  strength  model  (sensitivity  analysis) 


Input  Variable 

FSWP 

CAS 

BARCAP 

INT 

BAI 

OCA 

AAPk*AIRESC 

AAPk*SAPk 

AAPk*BARCAP 

FSWP*SAPk 

CAS*SAPk 

SAPk*INT 

BAI*OCA 

Total  Error 

Total  Corrected 

r2=  0.9873 


Sum  of  Squares 
1.5753 
15.2628 
3.5778 
17.8503 
.812812 
1.08781 
2.0503 
1.24031 
1.75781 
1.48471 
.877813 
.690313 
.331553 
2.5106 
54.0971875 


i2E _ Mean  Sq 

1  1.5753 

1  15.2628 

1  3.5778 

1  17.8503 

1  .8128 

1  1.0878 

1  2.0503 

1  1.24031 

1  1.75781 

1  1.48471 

1  .877813 

1  .690313 

1  .331553 

18  .115943 

31 


F-Ratio _ ErYalue 

11.3  .0035 

109  .0000 

25.7  .0001 

128  .0000 

5.83  .0266 

7.80  .0120 

14.7  .0012 

8.89  .0080 

12.6  .0023 

10.7  .0043 

6.29  .0239 

4.95  .0391 

23.8  .0001 


r2  (adjusted  for  degrees  of  freedom)  =  0.9781 


Each  of  the  terms  in  this  model  are  significant  at  the  5%  level.  The  r2  and 
adjusted  r2  values  are  quite  high.  Figure  4.2  contains  the  normal  plot  of  the  residuals. 
The  model  appears  to  account  for  most  of  the  significant  structure  in  the  data. 
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Normal  Probability  Plot 


Residuals 

Figure  4.2  Normal  plot  of  the  residuals  for  the  ground  strength  model  (sensitivity 

analysis) 

Figure  4.3  plots  the  residuals  versus  the  expected  value.  This  plot  also  indicates 
no  significant  structure  can  be  observed  to  remain  unaccounted  for  in  the  data. 
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Residual  Plot  for  Ground  Strength 


Predicted 

Figure  4.3  Plot  of  residuals  vs.  predicted  values  of  the  ground  strength  model 

(sensitivity  analysis) 

The  air  attrition  response  was  the  most  difficult  to  estimate.  The  air  attrition 
model  was  the  most  complex  of  the  original  experiment,  and  the  two  main  effects  added 
in  the  sensitivity  experiment  further  complicated  the  model.  Several  alternatives  were 
considered  in  developing  a  parsimonious  representation  of  the  simulation  response.  With 
respect  to  the  air  attrition  MOE  especially,  the  dividing  line  between  significant  and 
insignificant  factors  was  not  obvious.  Judgment  is  a  necessary  element  in  this 
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investigative  process.  The  largest  model  considered  included  the  mean  and  30  other 
terms.  The  ANOVA  for  this  model  is  presented  in  Table  4.3. 

Table  4.3  ANOVA  for  air  attrition 


Note  the  summary  measures  in  the  upper  right  comer  of  the  table.  Mallow's  Cp 
statistic  requires  an  estimate  of  the  variance  of  the  process  to  be  modeled.  The  variance 
used  (37.9)  is  the  average  of  the  10  replication  variances  computed  for  each  of  the  32 
design  points.  Also  note  the  critical  F  statistic  values  on  the  right  side  of  the  table.  They 
provide  the  threshold  criteria  for  determining  the  significance  of  individual  terms  in  the 
model.  Obviously  some  of  these  terms  are  not  significant. 
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A  useful  means  of  comparing  the  candidate  models  is  to  plot  the  mean  square 
error  as  the  number  of  terms  in  the  model  increases.  Figure  4.4  shows  a  plot  of  various 
air  attrition  models  versus  the  mean  square  enor  of  each  model.  Figure  4.4  helps  to 
identify  when  enough  terms  have  been  added  and  additional  terms  only  model  the  noise. 
It  is  possible  to  add  terms  that  improve  the  fit  for  this  particular  data  set  which  really 
have  nothing  to  do  with  the  process.  Any  model  for  air  attrition  involving  more  than  23 
terms  is  certainly  overfitting  the  data.  Any  model  with  less  than  1 1  terms  will  have  an 
MSE  greater  than  40,  whereas  the  average  sample  variance  for  air  attrition  was  37.9. 


Air  Attrition 


#  of  term  in  the  modal 


Figure  4.4  Graph  of  the  MSE  vs.  #  of  terms  in  the  model 

Table  4.4  shows  the  ANOVA  for  the  first  candidate  model  for  air  attrition.  It 
contains  23  terms  including  the  constant  term.  The  three  candidate  models  were  selected 
fi-om  those  1 1  to  23  terms.  Model  one  had  the  highest  r2  value  but  some  of  the  terms 
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have  F-ratios  of  less  than  3.36.  For  this  model,  any  term  with  an  F-ratio  of  less  than  3.36 
fails  the  significance  test  at  the  10%  level. 


Table  4.4  Model  1  for  air  attrition  (sensitivity  analysis) 
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0 
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Table  4.5  is  the  ANOVA  table  for  the  second  candidate  model  for  air  attrition. 
Since  it  has  fewer  terms  than  the  previous  model,  its  value  is  lower.  Each  of  the  terms 
in  this  model  passes  the  F-test  for  significance  at  the  10%  level.  But  not  at  the  5%  level. 
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Table  4.5  Model  2  for  the  air  attrition  (sensitivity  analysis) 


Table  4.6  is  the  ANOVA  for  the  third  candidate  air  attrition  model  and  has  the 
fewest  nvimber  of  terms  considered.  All  the  terms  are  also  significant  at  the  10%  level. 
This  model  has  two  advantages  over  the  two  previous  models.  It  is  simpler  and  the 
estimate  for  the  variance  of  the  air  attrition  process  matches  the  average  variance  of  the 
TAC  THUNDER  runs. 
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Table  4.6  Model  3  for  air  attrition  (sensitivity  analysis) 


3*1 :  VJ 1 :  :  1 1 1  (•]  k'  I  tci  4 kl  1  i  1  tn  tV* 

■■■■ 

IHHI 

IBB! 

^bhi 

DF 

l,». 

B^Hi 

^ra.eo5 

1 

R-SQ: 

I2jjy2j2^ 

310.006 

1 

310.006 
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IBI^B 

TDiBff 

0 

0^  0 

y2jjmjj2 

V: 

I22i2i;2l^ 

1576.411 

1 

1576.411 

41.01969 

■BIH 

1 

ir.s4 

F:BARCAP 

0 

0 

^0 

fofterm: 

11 

4T:405 

0 

0 

0 

3545 

0 

0 

0 

1 
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0 

0 

0 

0 

0 

0 

0 

0 

0 
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0 

21 
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T52 

0 

0 

0 

20 

237 
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0 

19 
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4^ 
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0 

16 
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0 
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1 
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1 
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0 
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0 

EF 
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EG 
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■■■ 
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The  third  model  has  a  mean  square  error  of  38.4.  This  value  agrees  nicely  with 
37.9,  which  is  the  estimate  from  the  10  repetitions  at  each  of  the  32  design  points.  Model 
three  was  chosen  because  the  MSE  is  approximately  equal  to  the  variance  of  the  data.  All 
of  its  terms  are  significant  at  the  10%  level. 

The  ANOVA  tables  show  which  two  factor  interactions  are  aliased  with  other  two 
factor  interactions.  Only  one  of  each  confounded  pair  is  believed  to  be  potentially 
significant.  Interactions  included  in  the  first  design  and  shown  to  be  insignificant  were 
assumed  to  continue  to  be  insignificant.  By  allowing  this  two  factor  aliasing,  the 
significant  two  factor  effects  could  be  estimated  without  increasing  the  resolution  of  the 
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experiment  to  a  resolution  five  design.  A  resolution  five  design  would  have  required 
twice  as  many  design  points  but  no  two  factor  interactions  would  have  been  aliased  with 
any  other  two  factor  interaction. 

The  equation  resulting  from  the  third  model  is: 

AirAtr  =  10.95  +{-\m\e-^.\Q2'iFSWP-.QmOCA+.mMIRESC)SAPk+2.QU*AAPk+ 
2015FSWP-.Q06A99CAS+.\miNT+.\99SOCA-.m22AlINT*OCA- 
m2AlFSWP*AIRESC+.  1 Q2UIRESC  (Eq  4-3) 

Figure  4.5  shows  a  plot  of  the  residuals  of  this  air  attrition  model  plotted  against 
the  predicted  value.  No  significant  pattern  was  observed  in  Figure  4.5  The  presence  of 
such  a  pattern  would  be  evidence  in  favor  of  a  model  with  more  terms. 
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Residual  Plot  for  Air  Attrition 


0  10  20  30  40  50 

Predicted 

Figure  4.5  Plot  of  the  residuals  vs.  the  predicted  values  for  the  Air  Attrition 
model  (sensitivity  analysis) 

Figure  4.6  shows  a  normal  probability  plot  of  the  residuals  of  the  air  attrition 
model.  The  presence  of  data  points  which  do  not  lie  along  the  line  indicates  that  some 
additional  structure  remains  unexplained  by  the  current  model. 
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Normal  Probability  Plot 


-11  -7  -3  1  5  9  13 

Residuals 

Figure  4.6  Normal  plot  of  the  residuals  of  the  Air  Attrition  model  (sensitivity 

analysis) 

The  model  selected  contains  the  most  significant  terms  and  predicts  a  variance 
approximately  equal  to  the  estimate  of  the  variance  foimd  in  the  data.  However,  the 
difficulties  in  selecting  a  "good"  air  attrition  model  indicate  that  the  air  attrition  process  is 
complex. 
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Analysis 


The  three  models  developed  from  the  sensitivity  experiment  form  the  basis  for  the 
analysis  of  the  relationship  between  the  various  outputs  and  the  two  operational 
uncertainty  variables.  The  partial  derivative  of  each  model  with  respect  to  an  operational 
imcertainty  variable  indicates  how  that  MOE  tends  to  change  as  the  imcertainty  variable 
increases.  The  partial  derivatives  represent  the  response  of  the  MOE  to  changes  in  the 
uncertainty  variables;  an  important  aspect  in  analyzing  the  behavior  of  TAG  THUNDER. 
Intuitively,  it  is  expected  that  as  the  opponents  become  more  capable,  their  results  should 
improve. 

The  partial  derivative  of  the  PLOT  with  respect  to  AAPk  is  : 

^  =  -2.05OCA  +  2.09FSWP+.0435BARCAP  (Eq  4-4) 

OCA  missions  become  more  effective  as  the  opponents  aircraft  become  more 
lethal  in  air-to  air  engagements.  Historically,  it  is  easier  to  kill  aircraft  on  the  ground  than 
in  air-to-air  combat.  FSWP  and  to  a  lesser  extent  BARCAP  missions  become  less  useful 
in  the  sense  that  they  generate  more  losses  per  mission  than  previously.  It  is  interesting 
to  note  that  there  is  not  a  significant  decrease  in  the  effectiveness  of  CAS  as  the  AAPk 
increases. 

The  partial  derivative  of  the  PLOT  with  respect  to  SAPk  is  : 

^  =  -61.4  (Eq4-5) 

Figure  4.7  shows  the  PLOT  observations  grouped  by  the  low  and  high  values  of 
the  SAPk  and  arranged  in  ascending  order.  As  seen  in  Figure  4.7,  the  net  effect  of 
doubling  the  opponent's  SAPk  was  that  they  made  less  progress  on  the  ground!  This 
response  is  very  nonintuitive.  One  possible  explanation  lies  in  the  sortie  assignment 
algorithm  of  TAC  THUNDER.  The  increased  SAPk  value  might  change  the  way  TAC 
THUNDER  organized  the  strike  packages  which  attacked  ground  targets. 

If  the  increased  SAPk  value  (which  the  algorithm  uses!)  changed  the  way  the 
aircraft  were  assigned  into  packages  and  these  packages  assigned  to  targets,  then  perhaps 
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* 


these  new  assignments  were  more  effective  at  stopping  PLOT  movement.  Clearly 
something  unusual  is  going  on  here. _ 

PLOT  grouped  by  SAPk  setting 


Figure  4.7  FLOT  vs.  SAPk  values 


The  partial  derivative  of  the  ground  strength  with  respect  to  AAPk  is: 

=  -0.000176AIRESC  +.197 SAPk  -0.007S BARCAP 

(Eq4-6) 

AIRESC  and  BARCAP  both  become  less  effective  (for  maximizing  the  ground 
strength)  when  the  AAPk  values  increase. 


The  partial  derivative  of  the  ground  strength  with  respect  to  SAPk  is: 

=  -^.0Q72FSWP  -  0.001 54C45  - 0.00306/iVr  +  0.\97 AAPk  (Eq  4-7) 


As  the  opponent's  SAPk  value  increases,  the  contribution  made  by  each  FSWP, 


CAS,  and  INT  sortie  is  reduced. 

The  partial  derivative  of  the  air  attrition  with  respect  to  SAPk  is: 


»  =  2.0125 


(Eq  4-8) 


When  the  opponent's  AAPk  value  changed  from  the  low  setting  to  the  high 
setting,  an  average  of  four  additional  aircraft  were  lost. 

The  partial  derivative  of  the  air  attrition  with  respect  to  SAPk  is: 

=  -1.0316  +  0.102F5PPP  -0.0833(9C4+.07033^//J£5C  (Eq  4-9) 

As  the  SAPk  value  increases,  the  value  of  OCA  missions  decreases.  This 
response  is  reasonable  since  airfields  tend  to  be  heavily  defended  against  aircraft  attacks. 
In  contrast,  the  value  of  FSWP  and  AIRESC  missions  increases  as  the  enemy's  SAPk 
increases.  It  may  be  that  these  aircraft  are  more  survivable  against  surface-to-air  threats. 
If  attacks  are  made  against  these  sorties  instead  of  other,  more  vulnerable,  sorties,  then 
the  average  number  of  aircraft  shot  down  would  be  reduced. 


Worst  Case  Analysis 

This  section  considers  how  the  optimal  apportionment  foimd  in  chapter  three 
performs  against  a  worst  case  combination  of  the  SAPk  and  AAPk  values.  Simply  setting 
SAPk  and  AAPk  to  their  highest  level  might  not  be  the  worst  case.  Worst  case  versions  of 
Equations  4-1  through  4-3  were  found  by  selecting  the  worst  value  for  each  term  in  the 
model.  For  example,  the  coefficient  associated  with  OCA  in  Equation  4-3  is  0.1995  +/- 
0.0833,  depending  on  the  value  of  the  SAPk  term.  For  air  attrition,  more  aircraft  are  lost 
as  a  coefficient  increases.  Therefore,  0.2828  was  the  value  used  for  the  OCA  coefficient 
in  the  worst  case  air  attrition  model.  Using  this  technique,  no  matter  what  the 
apportionment,  these  new  equations  will  represent  the  worst  possible  values. 
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PLOT  Worst  case: 


FLOT=  1 045+.0435BCAP-2.52CAS-4.9INT+.03FS  WP+.0426FS  WP*B  AI- 
.0192INT*OCA-.050BAl*OCA+4.47OCA-.4765BAI  (Eq  4-10) 

Ground  Strength  Worst  case: 

Grd  Str  =  85.38  -.0001757AIRESC-.0144FSWP+.0049CAS+.001176BARCAP- 
.000489BAI-.005 1 10CA+.000248BAI*OCA+  .306INT)  (Eq  4-1 1) 

Air  Attrition  Worst  case: 

Air  Atr  =  14  +.3098FSWP-.006499CAS+.1009INT+.28280CA- 
.002242INT*OCA-.00247FSWP*AIRESC+.  1 74AIRESC  (Eq  4-12) 

Optimizing  these  equations  for  their  respective  MOEs  yields  the  best  worst  case 
apportionment  for  each  MOE.  Figures  4.8  to  4.10  present  the  results  of  these 
apportionments.  Unlike  the  apportionment  in  chapter  three,  each  of  the  three  MOEs  has  a 
different  optimum  apportionment.  Figure  4.8  shows  the  predicted  PLOT  as  a  function  of 
the  number  of  aircraft  in  the  theater.  The  aircraft  have  been  optimized  for  each  of  the 
three  MOEs.  For  a  given  number  of  aircraft,  the  top  bar  represents  the  optimal 
apportionment  to  minimize  the  loss  of  aircraft.  The  middle  bar  represents  the  optimal 
apportionment  to  maximize  the  ground  strength.  The  bottom  bar  represents  the  optimal 
apportionment  to  minimize  the  PLOT  movement. 
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«  of  A/C 


Figure  4.8  FLOT  outcomes  as  a  function  of  the  apportionment  optimized  for  air 

attrition,  ground  strength,  and  FLOT 


Similarly  Figure  4.9  shows  the  predicted  ground  strength  value  of  the  three  sub  optimized 
air  apportionments. 
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GRO  STR  VS.  #  OF  A/C 


Figure  4.9  Ground  strength  comparisons  based  on  #  of  A/c  Figiue  4. 1 0  shows  the 

predicted  aircraft  attrition  as  a  function  of  the  number  of  aircraft  available. 
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Figure  4.10  Aircraft  attrition  as  a  function  of  #  of  aircraft  and  MOE  optimized 
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III 


Figures  4.8  to  4.10  show  how  optimizing  the  air  attrition  or  ground  strength 
causes  a  large  PLOT  result.  For  this  scenario  at  least,  a  low  casualty  rates  should  not  be 
the  primary  objective.  Minimizing  the  PLOT  movement  does  not  result  in  a  large 
increase  in  air  or  ground  losses. 

The  sensitivity  models  were  compared  to  the  baseline  models  by  substituting  -1 
for  the  SAPk  and  AAPk  terms  in  Equations  4-1, 4-2,  and  4-3.  The  predicted  values  for 
the  three  MOE's  were  generated  for  26  design  points.  These  design  points  were  the  16 
used  in  chapter  three  and  the  eight  design  points  in  chapter  four  that  had  SAPk  and  AAPk 
settings  of  -1 .  Ideally,  the  predictions  of  the  baseline  models  would  be  highly  correlated 
with  the  predictions  of  the  sensitivity  models.  The  correlations  between  the  baseline 
models  and  the  sensitivity  models  for  the  air  attrition,  ground  strength,  and  PLOT  MOE's 
were  .52,  .61,  and  .76,  respectively.  Appendix  D  contains  plots  of  the  baseline  models' 
predictions  versus  the  sensitivity  models'  predictions.  No  nonlinear  relationship  was 
observed  in  these  plots.  Several  interesting  features  were  observed  but  the  timing  of  this 
research  precluded  further  investigation.  Apparently,  there  were  significant  differences 
in  TAG  THUNDER'S  responses  that  were  not  accoimted  for  in  the  SAPk  and  AAPk 
terms. 

Conclusion; 

CAS  missions  proved  beneficial  under  the  range  of  operational  uncertainty 
variables.  No  other  mission  category  was  uniformly  beneficial  under  an  absolute  worst 
case  scenario.  Interdiction  could  also  help  the  PLOT  MOE  while  only  increasing  the  air 
attrition  slightly.  In  view  of  these  results,  the  baseline  apportionment  found  in  chapter 
three  is  still  valid  and  fairly  robust.  After  these  two  missions,  the  baseline  added  a  mix  of 
air  escort  and  interdiction  missions.  Under  the  worst  case  analysis,  these  missions 
reduced  the  overall  effectiveness  of  the  apportionment. 
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Response  surface  models  allow  a  quick  response  to  what-if  questions  without  re¬ 
running  a  time-consuming  model.  As  long  as  the  region  of  interest  has  been  covered  in 
the  design,  RSM  gives  the  analyst  insight  into  the  behavior  of  the  model  and  a  better 
ability  to  respond  to  new  questions  by  the  decision  maker. 

Such  insight  into  TAC  THUNDER'S  behavior  would  be  quite  useful  in  the 
verification,  validation,  and  accreditation  (VV&A)  process.  A  user  (or  designer)  of  a 
model  wants  to  know  that  the  model  behaves  in  a  reasonable,  rational  way.  It  must 
simulate  the  process  in  a  manner  appropriate  for  the  intended  application. 

Several  interesting  observations  were  made  by  analyzing  how  TAC  THUNDER 
responds  to  increases  in  the  probability  of  kill  values.  When  the  Pk  values  for  the  surface 
to  tiir  threats  were  increased  for  the  opponents,  the  opponents  had  less  success  with  the 
PLOT  MOE.  Intuitively  one  would  expect  the  opponents  to  make  more  progress.  One 
possible  explanation  is  that  the  sortie  generation  algorithm  used  the  change  in  Pk  values 
to  change  the  way  the  sorties  were  organized  and  flown.  The  goal  of  the  sensitivity 
experiment  was  to  observe  the  effect  on  the  baseline  case,  given  that  the  opposing  force 
was  more  effective  than  anticipated.  If  the  algorithm  behaves  in  such  a  manner,  it  is 
impossible  to  change  how  well  the  opponents  fight  without  changing  otir  perception  of 
their  capabilities.  Another  possibility  is  that  there  could  be  a  problem  with  the  algorithms 
that  use  the  S  APk  values.  One  experiment  does  not  prove  anything,  but  it  does  identify 
an  area  of  concern.  Additional  work  should  be  done  to  understand  how  TAC  THUNDER 
behaves.  It  is  possible  that  our  intuition  is  wrong  and  these  results  might  give  us  better 
insight  into  theater  level  combat.  That  is  one  of  the  primary  purposes  of  all  models:  to 
act  as  a  catalyst  for  insight. 


V.  Conclusions  and  Recommendations 


Summary 

The  pvupose  of  this  thesis  was  to  demonstrate  that  R5M  techniques  could  be  used 
to  optimize  the  air  apportionment  in  a  TAC  THUNDER  scenario.  An  unclassified 
scenario  was  developed  and  its  air  apportionment  optimized  for  three  MOEs:  PLOT, 
ground  strength,  and  air  attrition.  The  screening  design  provided  sufficient  data  to  model 
the  relationship  between  the  air  apportionment  and  the  three  MOEs.  The  metamodels 
which  quantified  the  relationship  between  the  model  inputs  and  the  MOEs  were 
optimized  with  constraints  appropriate  to  the  study.  These  metamodels  allow  an  analyst 
to  quickly  answer  what-if  questions  fi'om  the  decision  maker  as  long  as  the  quesL..as 
concern  the  region  bounded  by  the  original  design  space. 

In  the  first  phase  of  this  investigation  (chapter  three),  one  apportionment 
optimized  all  three  MOE's.  The  baseline  apportionment,  in  rank  order,  is  as  follows: 

1  -  Close  Air  Support 

2  -  Interdiction 

3  -  An  even  mix  of  Air  Escort  and  Fighter  Sweep  missions. 

The  Air  Escort  and  Fighter  Sweep  missions  reduced  the  air  attrition  while  having  no 
significant  effect  on  the  other  two  MOEs. 

The  second  phase  in  this  investigation  examined  the  sensitivity  of  the  response 
surfaces  to  two  operational  uncertainty  variables.  Close  Air  Support  proved  to  be  useful 
in  all  cases  for  all  three  MOEs.  The  response  of  the  three  MOEs  to  changes  in  the 
Surface-to-Air  Probability  of  Kill  (SAPk)  value  was  quite  interesting.  As  the  opponents 
S  APk  increased,  they  made  less  progress  on  the  ground  even  though  they  did  shoot  down 
more  of  our  aircraft.  An  explanation  for  this  result  needs  to  be  foimd.  This  is  an 
excellent  example  of  how  RSM  can  provide  understanding  of  the  behavior  of  complex 


5-1 


simulations  thereby  supporting  the  verification,  validation,  and  accreditation  (VV«&A) 
process. 

Recommendations  for  Follow-On  Efforts 

It  is  difficult  (and  incorrect)  to  generalize  with  data  from  only  one  phase  of  one 
scenario.  Additional  study  needs  to  be  done  on  other  phases  of  this  scenario.  Similar 
studies  using  different  scenarios  would  help  to  generalize  the  results  and  support  efforts 
to  use  TAG  THUNDER  to  examine  AF  doctrine.  The  sensitivity  of  the  air  apportionment 
to  changes  in  the  opponent's  air  apportionment  needs  to  be  studied.  The  new  Theater- 
Level  Model,  being  developed  at  the  Naval  Post-Graduate  School,  uses  a  simpler  air 
apportionment  that  would  be  ideal  for  studying  the  air  apportionments  of  opposing  forces 
using  RSM  and  game  theory.  Classified  studies  could  be  done  to  see  if  the  results 
corroborate  this  study.  A  detailed  study  could  be  made  of  how  well  the  metamodels 
predicted  the  TAG  THUNDER  output. 

An  important  area  of  study  is  multi-phase  operations.  Is  it  appropriate  to  optimize 
each  phase  of  a  campaign  separately?  Or,  does  the  entire  campaign  need  to  be  optimized 
simultaneously?  How  can  a  multi-phase  campaign  be  optimized  with  limited  time  and 
computing  resources? 

Finally,  VV&A  efforts  on  combat  models  could  incorporate  RSM  techniques.  It 
is  difficult,  if  not  impossible,  to  prove  that  a  model  "correctly"  models  theater-level 
warfare.  Aggregation  and  assumptions  are  fundamental  components  of  a  theater-level 
combat  simulation.  Identifying  the  relationship  between  the  inputs  and  outputs  of  a 
complex  simulation  should  provide  evidence  that  the  model  is  working  as  expected  or 
that  the  models  behaves  in  unexpected  ways.  In  that  case,  either  the  model  or  our 
expectations  would  need  to  be  changed. 

Stochastic  annealing  proved  an  effective  way  to  match  the  variables  to  the  factors 
in  the  experimental  design  stage.  This  technique  may  allow  experiments  with  large 
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numbers  of  variables  to  reduce  the  number  of  design  points  needed;  if  some  of  the 
factors  can  be  considered  negligible,  then  a  design  that  does  not  alias  any  of  the 
nonnegligable  terms  may  be  sufficient.  Additional  work  needs  to  be  done  In  this  area. 

ConclusiQns 

This  thesis  has  demonstrated  that  RSM  techniques  can  be  used  to  optimize  the  air 
apportionment  in  a  TAC  THUNDER  scenario.  It  is  limited  to  those  cases  where  the 
number  of  variables  can  be  reduced  to  a  manageable  level,  such  as  was  done  here  by 
grouping  the  aircraft  and  optimizing  only  one  phase  of  a  campaign.  Since  the  best  use  of 
force  is  required  to  study  tradeoffs  between  force  mixes,  this  technique  could  support 
many  combat  effectiveness  studies  and  acquisition  decisions. 
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APPENDIX  A 


This  appendix  provides  a  description  of  the  TAC  THUNDER  scenario  used  in  this 
thesis  effort.  It  will  describe  the  order  of  battle  for  both  sides  including  units,  squadrons, 
number  and  type  of  aircraft. 

Those  files  modified  from  the  standard  unclassified  Middle  East  scenario  are  also 
provided. 


The  order  of  battle  was: 


UNIT 

TANKS 

A£C 

HELP  ARTY  INF 

AIRDEF 

1  St  Saudi  Div 

348 

318 

42 

132 

1000 

12 

82nd  Airborne 

0 

270 

42 

54 

1500 

12 

24th  Mech  Inf  Div 

290 

270 

42 

126 

1000 

12 

IStMEF 

348 

ILS 

42 

132 

12 

TOTAL 

986 

1176 

168 

444 

4500 

48 

AIRCRAFT: 

F-15  36-216 

F-15E  0-194 

A-10  0-216 

EFlll  8-30 

Fill  18-72 

WEASEL  0-24 


IRAQI  FORCES: 


mu 

TANKS 

APC 

HELP.  ARTY  INK 

AD  GUN  SAM 

ARMOR  DIV  (x7) 

325 

250 

12 

150 

1000 

16 

8 

MECH  DIV  (x3) 

225 

650 

12 

150 

1200 

16 

8 

INF  DIV  (x8) 

2Q 

m 

JL 

150 

3000 

Ifr 

-Q 

TOTAL 

3670 

4900 

120 

2700 

34600 

288 

80 

AIRCRAFT: 

MIG29  25 

MIG23  100 

MIG21  150 

MIRAGE  FI  125 
SSU25  50 

AIR  DEFENSE  COMPLEXES :  1 0  S A-3  SITES 
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The  sortie  rates  used  included  a  maximum  surge  rate  for  the  first  six  days  of  the 


conflict  and  a  maximum  sustained  rate.  These  sortie  rates  by  aircraft  were: 


Al£ _ DAY.rN.THEATER.  .AUTH. 


A-10 

1.00 

3.00 

6.00 

2.00 

RF-4 

1.00 

2.50 

6.00 

1.50 

F-111 

1.00 

2.00 

6.00 

1.20 

F-15 

1.00 

2.50 

6.00 

2.20 

MIG-23 

1.00 

3.00 

6.00 

1.20 

MIRAGE  F-1 

1.00 

3.50 

6.00 

2.70 

MIG-21 

1.00 

3.00 

6.00 

1.20 

MIG-29 

1.00 

3.00 

6.00 

2.50 

SU-25 

1.00 

2.20 

6.00 

.80 

OTY.SORT/DAY  ..AC.MAX.SORT/DAY 
4.00 
3.00 
3.00 
2.00 

2.50 

1.50 
3.00 

2.50 

3.00 

1.20 

4.00 

2.70 

3.00 

1.20 

4.00 

2.70 

2.20 

.80 
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APPENDIX  B 


This  appendix  provides  a  description  of  the  simulated  annealing  code  developed 
in  this  thesis  effort.  The  code  is  also  provided. 

BACKGROUND 

High  resolution  designs  for  experiments  with  a  few  variables  are  simple  to  design 
and  carry  out.  As  the  number  of  variables  increase,  the  number  of  individual  experiments 
needed  to  achieve  the  same  level  of  resolution  increases  rapidly.  With  10  variables,  a 
fractional  factorial  design  requires  64  runs  to  achieve  resolution  five.  If  some  of  the  two 
variable  interactions  are  known  (or  assumed)  to  be  insignificant,  then  a  smaller  design 
can  be  used  where  none  of  the  significant  two  variable  interactions  are  confounded  with 
any  of  the  other  two  variable  interactions.  This  allows  a  reduction  in  the  resources 
required  to  carry  out  the  experiment. 

Each  variable  in  the  experiment  must  be  assigned  to  one  of  the  columns  of  the 
experimental  design  matrix.  For  a  given  design,  the  confounding  of  each  factor  is  fixed. 
For  example,  Factor  A*Factor  B  is  confounded  with  Factor  I*Factor  J.  If  the  variables 
are  assigned  to  factors  A,  B,  I,  and  J  such  that  AB  and  IJ  are  not  both  significant,  than  this 
confounding  is  not  a  problem.  The  number  of  ways  10  variables  can  be  assigned  to  10 
factors  is  10!  or  approximately  4  million  combinations. 

Simulated  Annealing 

Simulated  annealing  is  a  recent  technique  inspired  by  the  physical  process  in 
metals.  This  technique  uses  a  temperature  fimction  that  is  analogous  to  the  temperature 
in  the  cooling  metal.  As  the  temperature  drops,  the  atoms  in  the  metal  tend  to  arrange 
themselves  in  lower  energy  states.  In  this  implementation,  the  energy  state  is  the  value  of 
the  objective  fimction. 
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Each  variable  pair  represents  an  important  effect.  Weights  are  assigned  to  each 
pair  that  correspond  to  the  magnitude  of  their  significance.  These  weights  are  assigned 
by  the  experimenter  based  on  his  knowledge  of  the  process  under  study  and  previous 
experiments.  Each  assignment  of  variables  to  factors  has  a  value  equal  to  the  sum  of  the 
weights  of  all  the  confounded  pairs.  If  none  of  the  important  variable  pairs  are 
confounded  with  other  variable  pairs,  then  the  value  of  that  assignment  is  zero.  A  value 
of  zero  is  the  best  possible  value.  It  may  not  be  possible  to  deconflict  every  variable  pair. 
The  researcher  could  accept  a  sufficiently  low  value  if  appropriate. 

The  Code 

The  following  is  a  description  of  each  section  of  the  code: 

1-90  Define  the  variables. 

1 00- 1 90  Define  the  initial  assignment 

200-405  Define  the  important  variable  pairs. 

4 1 0-  500  Defme  the  weights  associated  with  each  important  variable  pair. 

1000-3390  Check  each  set  two  factor  interactions.  If  more  than  one  is  important  then  add  the  sum  of 
their  weights  to  the  objective  function. 

3400-3490  Implement  the  temperature  function  that  controls  how  many  iterations  will  be  made  at 
one  temperature  setting.  Once  the  correct  number  of  iterations  have  been  made,  the 
temperature  decreases. 

4000-4005  If  the  test  assignment  has  a  lower  value  than  best  assignment  found  so  far,  it  becomes  the 
new  best  assignment. 

4010  If  the  test  assignment  is  accepted  as  the  new  current  assignment,  then  it  becomes  the  new 

current  assignment. 

5000-6000  Generate  a  new  test  assignment  by  perturbing  the  current  assignment.  First  variable 
assignments  are  switched.  Two  variables  are  selected  randomly  and  switched  repeatedly. 


The  number  of  exchanges  are  a  function  of  the  temperature.  At  high  temperatures  the 
mean  number  of  switches  is  large.  As  the  temperature  falls,  the  average  perturbation 
becomes  smaller. 

1 0000- 1 0200  Subroutine  which  updates  the  best  solution  found. 

1 1 000- 1 1200  Subroutine  which  updates  the  current  solution. 

20000-20200  Print  the  best  solution  found  by  the  end  of  the  search. 


5  RANDOMIZE 

10  REM  Program  for  assigning  variables  for  RSM  and  Ending  the  number  of 
20  REM  confounded  terms. 

30  DIM  V(40):  REM  These  are  the  values  for  the  cross  terms. 

50  DIM  TF$(40):  REM  these  are  the  important  cross  terms. 

60  DIM  A$(10);  REM  these  are  the  factors. 

70  DIM  P$(10):  REM  these  are  the  best  solution  found. 

80  DIM  CS(IO):  REM  these  are  the  current  solution. 

90  TEMP  =  5:  REM  the  initial  temperature. 

100  REM  Initial  settings  for  10  variables  to  the  10  factors 
1 10  A$(l)  =  "5":  A$(2)  =  "9":  A$(3)  =  "3": 

120  A$(4)  =  "8":  A$(5)  =  "1":  A$(6)  =  "0" 

130  A$(7)  =  "4":  AS(8)  =  "2":  A$(9)  =  "7":  TC  =  999 
140  A$(10)  =  "6":  TB  =  999:  REM  TB  is  the  best  T  value  to  date 
150  FORi  =  1  TO  10 
160  P$(i)  =  A$(i) 

170  NEXT! 

200  REM  Defining  the  interaction  terms  TFS 

210  REM  aa=l,  ag=2,  ga=3  oca=4  eair=5  barcap=6  fswp=7  ^  iit=9  bai=0 
220  TF$(1)  =  "49":  TF$(2)  =  "94" 

230  TF$(3)  =  "57":  TF$(4)  =  "75" 

240  TF$(5)  =  "89":  TF$(6)  =  "98" 

250  TF$(7)  =  "14":  TF$(8)  =  "41" 

260  TFS(9)  =  "15":  TF$(10)  =  "51" 

TF$(12)  =  "6r' 

TF$(14)  =  "71" 

TF$(16)  =  "81" 

TF$(18)  =  "91" 

TF$(20)  =  "01" 

TF$(22)  =  "21" 

TF$(24)  =  "31" 

TF$(26)  =  "42" 

TF$(28)  =  "62" 

TF$(30)  =  "43" 

TF$(32)  =  "73" 

TF$(34)  =  "83" 

TF$(36)  =  "39": 

TF$(38)  =  "30":  REM  remove  03  and  30  and  0  value  sol  exists 
TFS(40)  =  "**" 
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270  TF$(11)  =  ’T6": 
280  TF$(13)  =  "17": 
290TF$(15)  =  "18": 
300TF$(17)  =  'T9": 
310TF$(19)  =  "10": 
320  TF$(21)  =  "12": 
330  TF$(23)  =  "13": 
340  TF$(25)  =  "24": 
350  TFS(27)  =  "26": 
360  TF$(29)  =  "34": 
370TF$(31)  =  "37": 
380  TF$(33)  =  "38": 
390  TF$(35)  =  "93": 
400  TF$(37)  =  "03": 
405  TF$(39)  = 


410  REM  Now  to  enter  the  values  for  the  cross  terms. 

420  V(l)=  100:  V(2)  =  V(l) 

421  V(3)  =  99:  V(4)  =  V(3) 

422  V(5)  =  98:  V(6)  =  V(5) 

423  V(7)  =  95:  V(8)  =  V(7) 

424  V(9)  =  80:  V(10)  =  V(9) 

425  V(ll)  =  58:  V(12)  =  V(11) 

426  V(13)  =  28:  V(14)  =  V(13) 

427  V(15)  =  25:  V(16)  =  V(15) 

428  V(17)  =  23:  V(18)  =  V(17) 

429  V(19)  =  21:  V(20)  =  V(19) 

430V(21)  =  20:  V(22)  =  V(21) 

431  V(23)=  12:  V(24)  =  V(23) 

432  V(25)  =  8:  V(26)  =  V(25) 

433  V(27)  =  7:  V(28)  =  V(27) 

434  V(29)  =  5:  V(30)  =  V(29) 

435  V(31)  =  4:  V(32)  =  V(31) 

436  V(33)  =  3:  V(34)  =  V(33) 

437V(35)  =  2:  V(36)  =  V(15) 

438V(37)  =  2:  V(38)  =  V(17) 

439  V(39)=  1:  V(40)  =  V(19) 

1000  REM  now  to  count  the  confounding,  each  100  is  a  constraint 
1010T  =  0:xl  =0 
1 100  FOR  i  =  1  TO  38 

1 1 10  IF  A$(l)  +  AS(2)  =  TF$(i)  THEN  xl  =  xl  +  1 :  Vtemp  =  Vtemp  +  V(i) 

1 120  IF  A$(9)  +  AS(IO)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

1160  NEXT! 

1170  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
1180x1  =0:  Vtemp  =  0 
1200  FOR  i  =  1  TO  38 

1210  IF  A$(l)  +  AS(3)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

1220  IF  A$(8)  +  A$(10)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

1260  NEXT  i 

1270  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
1280  xl  =  0:  Vtemp  =  0 
1300FORi  =  1  TO  38 

1310  IF  A$(l)  +  A$(4)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

1320  IF  A$(7)  +  AS(IO)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

1360  NEXT  i 

1370  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
1380  xl  =  0:  Vtemp  =  0 
1400  FOR  i  =  1  TO  38 

1410  IF  A$(l)  +  A$(5)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

1420  IF  A$(6)  +  A$(10)  =  TFS(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

1460  NEXT  i 

1470  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
1480  xl  =  0:  Vtemp  =  0 
1500FORi  =  lTO38 

1510  IF  A$(l)  +  A$(6)  =  TF$(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

1520  IF  A$(5)  +  A$(10)  =  TF$(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

1560  NEXT! 

1570  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
1580  xl  =  0:  Vtemp  =  0 
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FOR  i  =  1  TO  38 

IF  A$(l)  +  A$(7)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  A$(4)  +  AS(IO)  =  TFS(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

NEXT! 

IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
xl  =  0;  Vtemp  =  0 
FOR  i  =  1  TO  40 

IF  A$(l)  +  AS(8)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  A$(3)  +  AS(IO)  =  TFS(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

NEXTi 

IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
xl  =  0:  Vtemp  =  0 
FOR  i  =  1  TO  38 

IF  A$(l)  +  A$(9)  =  TFS(i)  THEN  xl  =  xl  +  1 :  Vtemp  =  Vtemp  +  V(i) 

IF  A$(2)  +  AS(IO)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

NEXTi 

IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
xl  =  0:  Vtemp  =  0 
FOR  i  =  1  TO  38 

IF  A$(l)  +  A$(10)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  A$(2)  +  A$(9)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  A$(3)  +  A$(8)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  A$(4)  +  A$(7)  =  TF$(i)  THEN  xl  =  xl  +  I:  Vtemp  =  Vtemp  +  V(i) 

IF  AS(5)  +  A$(6)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

NEXTi 

IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
xl  =0:  Vtemp  =  0 
FORi=  1  TO  38 

IF  A$(2)  +  A$(3)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  A$(8)  +  A$(9)  =  TF$(i)  THEN  xl  =  xl  +  I:  Vtemp  =  Vtemp  +  V(i) 

NEXTi 

IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
xl  =  0:  Vtemp  =  0 
FOR  i  =  1  TO  38 

IF  AS(2)  +  A$(4)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  A$(7)  +  A$(9)  =  TFS(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

NEXT  i 

IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
xl  =  0:  Vtemp  =  0 
FOR  i  =  1  TO  38 

IF  A$(2)  +  A$(5)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  AS(6)  +  A$(9)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

NEXTi 

IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
xl  =  0:  Vtemp  =  0 
FOR  i  =  1  TO  38 

IF  A${2)  +  A$(6)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

IF  A$(5)  +  A$(9)  =  TF$(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

NEXTi 

IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
xl  =  0:  Vtemp  =  0 
FOR  i  =  1  TO  38 

IF  A$(2)  +  AS(7)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 


B-5 


2520  IF  AS(4)  +  AS(9)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

2560  NEXT  i 

2570  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
2580  xl  =  0;  Vtemp  =  0 
2600  FOR  i  =  1  TO  38 

2610  IF  A$(2)  +  A$(8)  =  TF$(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

2620  IF  A$(3)  +  A$(9)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

2660  NEXT  i 

2670  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
2680  xl  =  0:  Vtemp  =  0 
2700  FOR  i  =  1  TO  38 

2710  IF  A$(3)  +  A$(4)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

2720  IF  AS(7)  +  A$(8)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

2760  NEXT  i 

2770  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
2780  xl  =  0:  Vtemp  =  0 
2800  FOR  i  =  1  TO  38 

2810  IF  A$(3)  +  A$(5)  =  TF$(i)  THEN  xl  =  xl  +  I:  Vtemp  =  Vtemp  +  V(i) 

2820  IF  A$(6)  +  A$(8)  =  TF$(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

2860  NEXT  i 

2870  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
2880x1  =0:  Vtemp  =  0 
2900  FOR  i  =  1  TO  38 

2910  IF  AS(3)  +  A$(6)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

2920  IF  A$(5)  +  A$(8)  =  TF$(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

2960  NEXT  i 

2970  IF  xl  >  I  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
2980  xl  =  0:  Vtemp  =  0 
3100FORi=  1  TO  38 

3 1 10  IF  A$(3)  +  AS(7)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

3 120  IF  A$(4)  +  A$(8)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

3 160  NEXT  i 

3 170  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
3180x1  =0:  Vtemp  =  0 
3200  FOR  i  =  I  TO  38 

3210  IF  A$(4)  +  A$(5)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

3220  IF  A$(6)  +  A$(7)  =  TFS(i)  THEN  xl  =  xl  +  1;  Vtemp  =  Vtemp  +  V(i) 

3260  NEXT  i 

3270  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 
3280x1  =0:  Vtemp  =  0 
3300  FOR  i  =  1  TO  38 

33 10  IF  A$(4)  +  A$(6)  =  TF$(i)  THEN  x  1  =  x  1  +  1 :  Vtemp  =  Vtemp  +  V(i) 

3320  IF  A$(5)  +  A$(7)  =  TF$(i)  THEN  xl  =  xl  +  1:  Vtemp  =  Vtemp  +  V(i) 

3360  NEXT  i 

3370  IF  xl  >  1  THEN  T  =  T  +  Vtemp:  REM  T  =  total  value  of  confounding  interactions 

3380  xl  =  0:  Vtemp  =  0 

3400  REM  coimtering  and  feedback  to  user 

3410  counter  =  counter  +  1 :  PRINT  TC; 

3420  IF  counter  =  30  THEN  TEMP  =  TEMP  *  .5:  PRINT  "TEMP=  TEMP:  counter  =  0 

3440  IF  TEMP  <  .05  THEN  GOTO  20000;  REM  These  lines  control  how  long  to  spend  at  one  temp 

3450  REM  and  how  quickly  the  temperature  falls.  3440  controls  when  to  stop. 

4000  IF  T  <  TB  THEN  C  =  0:  TB  =  T;  GOSUB  10000:  REM  updating  best  solution. 

4005  IF  T  =  TB  THEN  GOSUB  10000 
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4010  IF  T  <  TC  *  (TEMP  +  1)  THEN  GOSUB  1 1000 
5000  REM  need  to  generate  a  new  selection  of  variables. 

SOlOfl  =INT(RND*  10)  +  1 
5020  f2  =  INT(RND  •  10)  +  1 
5030  IF  n  =  f2  THEN  GOTO  5010 
5040  FOR  j  =  1  TO  10 

5050  A$0)  =  C$0) 

5060  NEXT  j 

5070  REM  resets  aS  to  current  CS  values 

5200  REM  swap  two  variable  assignments  between  factors  aS(fl)  and  aS(f2) 

5210  REM  tempS  is  a  temporary  variable 
5220  TEMPS  =  A$(f2) 

5230  A$(f2)  =  AS(fl) 

5240  A$(n)  =  TEMPS 

5232  flips  =  flips  +  RND 

5245  IF  flips  <  TEMP  +  .3  THEN  GOTO  5000 

5347  flips  =  0 

5250  REM  redo  calculations 
5260  GOTO  1000 
10000  REM  update  best  solution 
10010  FOR  i=  1  TO  10 
10020  P$(i)  =  A$(i) 

10025  IF  C  <  10  THEN  IF  TB  <  30  THEN  PRINT  A$(i), 

10030  REM  PRINT  A$(i), 

10040  NEXTi 

10045  IF  C  <  10  THEN  IF  TB  <  30  THEN  PRINT  "value  of  interactions  is  TB:  C  =  C  +  1 
10050  REVi  "The  value  of  confounded  interactions  is  TB 
10100  IF  TB  <  2  THEN  END 
101 10  RETURN 

1 1000  REM  updating  the  current  solution 
11100FORi  =  1  TO  10 
11110C$(i)  =  A$(i) 

11 120  NEXTi 
11130  TC  =  T 
1 1 140  RETURN 

20000  REM  routine  to  end  program 
20100  PRINT  :  "Best  solution  found:" 

20110FORi  =  1  TO  10 

20120  PRINT  "Factor i;  "should  be  assigned  variable  P$(i) 

20 130  NEXTi 

20140  PRINT  "The  value  of  the  interactions  is  ";  TB 
20150  END 
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APPENDIX  P 


This  appendix  contains  the  plots  used  to  compare  the  baseline  models  to  the 
sensitivity  models.  The  sensitivity  models  had  the  SAPk  and  AAPk  terms  equal  to  -1. 
The  PLOT  comparison  shows  a  reasonable  correlation  when  the  predicted  PLOT 
movement  exceeds  500km.  The  sensitivity  model  predicted  negative  PLOT  values  for 
design  points  that  the  baseline  model  predicted  a  PLOT  movement  of  approximately  50 
km. 
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The  ground  strength  comparison  showed  no  unusual  features. 
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The  air  attrition  comparison  showed  groupings  which  appeared  to  be  correlated. 
The  upper  left  data  points  all  had  the  maximum  number  of  CAS  missions.  Once  again, 
CAS  seems  to  have  a  significant  impact  on  the  air  attrition  process.  Further  investigation 
is  warranted  on  the  relationship  between  air  attrition  and  CAS  missions. 
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